AutoCompress writes deployment limits into pruning search: up to 6.57 points higher average accuracy than Minitron++ across three GPUs
Synopsis
LLM pruning usually targets parameter counts, but parameter count does not predict latency, throughput or memory, which depend on architecture, workload, hardware and serving engine. AutoCompress puts latency, throughput, parameter and memory constraints directly into differentiable architecture search, searching global dimensions and per-layer structure together and guided by cost models calibrated by measurement inside the target serving engine. Under the same deployment budgets and matched recovery training as Minitron++, it raises average accuracy across eight benchmarks by 6.57, 2.35 and 0.57 percentage points on RTX 5070, RTX 5080 and H100, using roughly one-half to one-third as many architecture-selection tokens.
Interpretation
The pruning target moves from a parameter budget to deployment constraints: given hardware, serving configuration, precision and workload, AutoCompress searches directly for a compressed architecture that meets latency, throughput or memory limits and outputs a standalone model. Earlier Minitron-style pipelines search within a user-defined search space and parameter budget, leaving practitioners to translate deployment requirements into pruning targets; parameter count itself does not reflect execution cost on the target hardware and serving engine. Across long-context local coding, batched datacenter serving and latency-sensitive in-game interaction, against a Minitron++ baseline using the same cost models and budgets, AutoCompress scores 6.57, 2.35 and 0.57 percentage points higher average accuracy on eight benchmarks on RTX 5070, RTX 5080 and H100.
The search decides global dimensions and per-layer structure together, so layers can keep different widths and layers can be removed outright. Minitron's search restricts layers to uniform dimensions, excluding architectures that allocate capacity differently across layers; AutoCompress places hidden size, Mamba head count and per-layer expert counts and feed-forward widths in one search space. On H100 both methods select the same hidden size and Mamba head count; Minitron++ uses uniform expert counts and intermediate sizes while AutoCompress varies both per layer, yielding 0.6% lower distillation loss, 0.57 percentage points higher average accuracy and 3.3 times fewer architecture-selection tokens.
Cost models measure per-layer runtimes inside the target serving engine and sum them, while memory is estimated item by item from weights, request cache, prefill workspace and runtime buffers. Estimating memory from weights and KV cache alone omits execution workspaces and runtime buffers; across six served RTX models such an estimate under-predicts memory by 31% to 43%. Explicit accounting reduces the error by roughly an order of magnitude; on RTX 5080, 60 runs covering 468 timing-table entries took about 3.5 GPU-hours, H100 used 180 runs and 24.8 GPU-hours, and RTX 5070 used 96 runs and 4.5 GPU-hours.
Perspective
The demonstrated scope is three concrete deployment targets: a game NPC on an RTX 5070 with a 4 GiB memory budget and 750 ms latency, a local coding agent on an RTX 5080 with 13.96 GiB and 11.5 s, and datacenter throughput of 7,330 tok/s per GPU on eight H100s; plus a case compressing Nemotron 3.5 Super from 120B to 74B to serve 128K context in BF16 on a single B200. It suits engineers working on LLM deployment and pruning.
In the large-scale case the compressed model's unweighted mean over nine benchmarks is 54.74 against 57.58 for the parent, a 2.85-point decrease; given the training cost, the Minitron++ baseline was not run for that case. Additive latency estimates can be affected by architecture-induced changes in kernel selection, cache behavior and overhead, and their accuracy was validated only on complete architectures excluded from timing-table construction.