Pruning calibration stops reading fixed text: decode-time activations lift Qwen3-32B writing score from 2.99 to 5.31

Pruning calibration no longer uses pre-collected fixed natural text; it uses the layer input activations the dense model produces token by token during autoregressive generation, keeping only decode steps and discarding prefill. Earlier training-free pruning methods collected calibration activations from fixed sequences such as C4 under teacher forcing, while at decode time the model is fed its own generated tokens, creating a distribution shift between the two. Across Llama-3.1-8B, Llama-3.3-70B and Qwen3-14B/32B, with the same SparseGPT backend and the same retained weight fraction, SparseDecoding scores higher than C4 calibration on overall WritingBench, for example raising Qwen3-32B from 2.99 to 5.31 under 2:4 sparsity.

The core linear layers during decoding behave more like sparse matrix-vector multiplication than sparse matrix-matrix multiplication, so a dedicated SpMV execution path is needed to turn sparsity into actual latency reduction. Existing sparse Tensor Core libraries target SpMM, reaching speedup during prefill but only a small fraction of dense throughput during decoding. Measured on NVIDIA A100 GPUs with the GPT-Fast framework, four N:M patterns give roughly 1.42x, 1.39x to 1.48x, 1.34x to 1.35x and 1.45x end-to-end decoding speedup on Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B and Qwen3-32B respectively.

The sparsity pattern is represented with bitmasks, so every row traverses all its nonzeros in a fixed number of steps. General sparse formats store a separate column index or local offset for each nonzero, which costs more metadata and gives rows different traversal lengths. At 50% sparsity the bitmask representation uses only 1 bit of metadata per nonzero, 32x and 8x less than int32 indices and uint8 offsets; each aligned 32-bit mask word holds exactly 16 set bits, fixing the per-row traversal count at compile time so the loop can be fully unrolled.

The report covers the training-free layer-wise pruning setting: calibration activations come from the pruned dense model's own autoregressive decode steps, prefill activations are discarded, and the solver stays an OBS-based backend such as SparseGPT or Wanda. Evaluation uses four instruction-tuned models, Llama-3.1-8B, Llama-3.3-70B and Qwen3-14B/32B, on WritingBench and ClassEval, on NVIDIA A100 80GB GPUs, with throughput measured through the GPT-Fast framework. It suits readers working on long-output inference deployment and post-training compression.

The authors report that the activation discrepancy rises rapidly during early autoregressive decoding and reaches at least part of each layer's eventual plateau within the first 12 decoding steps, then stays largely stable. The cross-model ablation shows that calibrating the 14B model on tokens generated by the larger 32B model scores below self-generated tokens, indicating that distributional agreement between calibration data and the pruned model is a contributing condition. These conclusions are limited to the evaluated models, sparsity patterns and benchmarks.