Sheldon Liang

LLM pruning usually targets parameter counts, but parameter count does not predict latency, throughput or memory, which depend on architecture, workload, hardware and serving engine. AutoCompress puts latency, throughput, parameter and memory constraints directly into differentiable architecture search, searching global dimensions and per-layer structure together and guided by cost models calibrated by measurement inside the target serving engine. Under the same deployment budgets and matched recovery training as Minitron++, it raises average accuracy across eight benchmarks by 6.57, 2.35 and 0.57 percentage points on RTX 5070, RTX 5080 and H100, using roughly one-half to one-third as many architecture-selection tokens.

LLM pruning usually targets parameter counts, but parameter count does not predict latency, throughput or memory, which depend on architecture, workload, hardware and serving engine. AutoCompress puts latency, throughput, parameter and memory constraints directly into differentiable architecture search, searching global dimensions and per-layer structure together and guided by cost models calibrated by measurement inside the target serving engine. Under the same deployment budgets and matched recovery training as Minitron++, it raises average accuracy across eight benchmarks by 6.57, 2.35 and 0.57 percentage points on RTX 5070, RTX 5080 and H100, using roughly one-half to one-third as many architecture-selection tokens.