4-bit LLM inference drops the calibration set: weight-error minimization doubles as activation scaling, matching SmoothQuant on MX and NV formats

A large language model's weights and activations can both be pushed to 4 bits without a batch of representative text for calibration. Activation quantization previously leaned on calibration data, as in SmoothQuant, which collects per-channel activation statistics from a calibration set to build its rescaling; the calibration-free rotation method QuaRot avoids that data but does not target activation-quantization error. Across the Llama and Qwen families, on WikiText-2 and C4 perplexity and on zero-shot accuracy for MMLU, HellaSwag, PIQA, ARC-Easy and WinoGrande, DualQuant matches or surpasses calibration-based SmoothQuant on most models and formats and clearly beats the calibration-free QuaRot and SINQ+, with especially large margins on MX and NV block formats.

The method gives each weight matrix both a row scale and a column scale, so quantization error is spread across two directions instead of one shared scale per row block. Standard block quantization assigns a single shared scale per row, so an outlier at one position inflates the scale shared by the whole block and the column dimension goes unaddressed. Turning on only the row scales or only the column scales each helps less than turning on both, and the two combine with roughly additive improvement, indicating they target complementary error sources.

The column scales work as activation rescalings because they are computed from the weights alone yet happen to attenuate the channels where activation outliers concentrate. SmoothQuant's scales come from activation statistics on a calibration set, so information flows from activations into weights; DualQuant's column scales come only from the static weight matrix. In an isolated experiment that holds the quantized weights fixed and compares activation signal-to-quantization-noise ratio, applying the column scales raises activation SQNR consistently across projections, with the largest gains on k_proj and v_proj, the projections where activation outliers are most pronounced.

The report covers calibration-free post-training weight-and-activation quantization: on the Llama and Qwen families, for the four 4-bit formats INT4, MXINT4, MXFP4 and NVFP4, measured by WikiText-2 and C4 perplexity and by zero-shot accuracy on MMLU, HellaSwag, PIQA, ARC-Easy and WinoGrande, with a real packed W4A4 inference path for Llama-3.1-8B validated on a single RTX 4090. It is aimed at people working on low-bit inference deployment and quantization algorithms.

The correlation between the column scales and SmoothQuant's calibration-derived scale varies by projection, and the paper states that this correlation is descriptive and does not imply equivalent quantization behavior. The SINQ+ results on MX and NV formats come from the authors' own extension, were not validated by the original method's authors, and should be treated as best-effort baselines. The real packed path is currently validated only for Llama-3.1-8B in the INT4 grouped setting.