# Low-bit quantization

Reasoning vision-language models must hold both multimodal context and a growing reasoning trace in their KV cache, and existing low-rank compression calibrates on static inputs, so its divergence from the dense model grows with generation length. TraceKV instead calibrates on reasoning traces the dense model generates itself, weights each KV state by its sensitivity to policy divergence before the SVD, and allocates ranks globally. Across four models including Qwen2.5-VL-7B, a 70% rank ratio with 4-bit quantization gives 73.41% overall accuracy versus 73.58% for the dense model, with the KV cache reduced 5.64 times.

Reasoning vision-language models must hold both multimodal context and a growing reasoning trace in their KV cache, and existing low-rank compression calibrates on static inputs, so its divergence from the dense model grows with generation length. TraceKV instead calibrates on reasoning traces the dense model generates itself, weights each KV state by its sensitivity to policy divergence before the SVD, and allocates ranks globally. Across four models including Qwen2.5-VL-7B, a 70% rank ratio with 4-bit quantization gives 73.41% overall accuracy versus 73.58% for the dense model, with the KV cache reduced 5.64 times.

Low-bit quantization cuts the inference cost of LLM agents but often makes them overlook task requirements or environmental conditions and take wrong steps in tool use. SteerDistill has a full-precision teacher locate the student's first behavioral divergence, issue a structured correction prompt, and refine that prompt using execution feedback until the revision passes local verification. The student then learns these corrections without guidance, and across five agentic benchmarks and three models under W4A16 its average score exceeds the strongest quantized baseline by 1.67–2.18 points and stays within one point of BF16.

Low-bit quantization cuts the inference cost of LLM agents but often makes them overlook task requirements or environmental conditions and take wrong steps in tool use. SteerDistill has a full-precision teacher locate the student's first behavioral divergence, issue a structured correction prompt, and refine that prompt using execution feedback until the revision passes local verification. The student then learns these corrections without guidance, and across five agentic benchmarks and three models under W4A16 its average score exceeds the strongest quantized baseline by 1.67–2.18 points and stays within one point of BF16.

Large language models store part of their factual knowledge in hash tables keyed by preceding token sequences, and low-bit quantization erases long-tail facts first, while giving more bits to frequently accessed rows does worse than choosing at random. GateRQ stores each row as a ladder of residual codes, computes each lookup's gate from a 1-bit code, and reads deeper levels only for the lookups the gate ranks highest. On DeepSeek-V4.1-Flash it cuts table memory from 23.6 to 3.04 GiB per GPU and raises vLLM throughput over FP8 tables in GPU memory by about 11% and about 31% on the two workloads, losing about 1 PopQA point at 1.50 bits per lookup.

Large language models store part of their factual knowledge in hash tables keyed by preceding token sequences, and low-bit quantization erases long-tail facts first, while giving more bits to frequently accessed rows does worse than choosing at random. GateRQ stores each row as a ladder of residual codes, computes each lookup's gate from a 1-bit code, and reads deeper levels only for the lookups the gate ranks highest. On DeepSeek-V4.1-Flash it cuts table memory from 23.6 to 3.04 GiB per GPU and raises vLLM throughput over FP8 tables in GPU memory by about 11% and about 31% on the two workloads, losing about 1 PopQA point at 1.50 bits per lookup.

When a multimodal large language model is quantized to low bit-widths, text and vision tokens must learn one shared channel-wise transform, and their gradients can point in opposite directions or differ wildly in magnitude. MCTQ splits the reconstruction objective by modality, then combines the two gradients into a single update direction through a minimum-norm convex combination, keeping one shared transform and adding no inference overhead. On Qwen2.5-VL-7B at W4A6 it reaches 66.26 average accuracy, 1.30 points above OmniQuant; Qwen2.5-VL-32B reaches 66.51 in the same setting.

When a multimodal large language model is quantized to low bit-widths, text and vision tokens must learn one shared channel-wise transform, and their gradients can point in opposite directions or differ wildly in magnitude. MCTQ splits the reconstruction objective by modality, then combines the two gradients into a single update direction through a minimum-norm convex combination, keeping one shared transform and adding no inference overhead. On Qwen2.5-VL-7B at W4A6 it reaches 66.26 average accuracy, 1.30 points above OmniQuant; Qwen2.5-VL-32B reaches 66.51 in the same setting.

The work introduces OSFP4: by analyzing a multiplicatively dithered FP4 quantizer it obtains a differentiable approximation of the matrix-product quantization error, then jointly optimizes a diagonal smoothing matrix and weight block scales per linear projection, with separate losses for RTN and GPTQ-style SIC rounding; on Llama-3.1-8B-Instruct the W4A4 configuration attains the highest average accuracy among the evaluated methods while retaining roughly 94-97% of vendor NVFP4 prefill throughput.

The work introduces OSFP4: by analyzing a multiplicatively dithered FP4 quantizer it obtains a differentiable approximation of the matrix-product quantization error, then jointly optimizes a diagonal smoothing matrix and weight block scales per linear projection, with separate losses for RTN and GPTQ-style SIC rounding; on Llama-3.1-8B-Instruct the W4A4 configuration attains the highest average accuracy among the evaluated methods while retaining roughly 94-97% of vendor NVFP4 prefill throughput.

To address the KV cache memory bottleneck that grows with recurrent depth in looped Transformers, the authors propose ResidualQuant, which uses the final-loop KV states as an anchor and represents the remaining loops as low-precision residuals, combined with least-squares scaling, rotations applied to the residuals, and loop-wise INT4/INT2 mixed precision; across Ouro-1.4B and Huginn-3.5B on mathematical reasoning and code generation benchmarks it improves the accuracy-memory tradeoff over rotation-based KV quantization, reducing theoretical KV storage by 80.7% relative to BF16 while retaining accuracy close to BF16 under mixed precision.

To address the KV cache memory bottleneck that grows with recurrent depth in looped Transformers, the authors propose ResidualQuant, which uses the final-loop KV states as an anchor and represents the remaining loops as low-precision residuals, combined with least-squares scaling, rotations applied to the residuals, and loop-wise INT4/INT2 mixed precision; across Ouro-1.4B and Huginn-3.5B on mathematical reasoning and code generation benchmarks it improves the accuracy-memory tradeoff over rotation-based KV quantization, reducing theoretical KV storage by 80.7% relative to BF16 while retaining accuracy close to BF16 under mixed precision.