# KV Cache Quantization

StrataFlux, from Cornell and Carnegie Mellon researchers, keeps low-bit weights and KV cache resident on the GPU and fetches only a small set of refinement bits from host RAM or SSD for the computation at hand, recovering accuracy while retaining quantization's memory and speed advantages. On an RTX 5090, 2-bit Llama-3.1-8B-Instruct weights drop WikiText-2 perplexity from 9.46 to 8.40 while keeping 95.4% of the quantized base's throughput; with 2-bit weights and KV, Qwen3-8B's AIME 2024/2025 accuracy rises from 11.0% to 47.1% at unchanged GPU memory use.

StrataFlux, from Cornell and Carnegie Mellon researchers, keeps low-bit weights and KV cache resident on the GPU and fetches only a small set of refinement bits from host RAM or SSD for the computation at hand, recovering accuracy while retaining quantization's memory and speed advantages. On an RTX 5090, 2-bit Llama-3.1-8B-Instruct weights drop WikiText-2 perplexity from 9.46 to 8.40 while keeping 95.4% of the quantized base's throughput; with 2-bit weights and KV, Qwen3-8B's AIME 2024/2025 accuracy rises from 11.0% to 47.1% at unchanged GPU memory use.

SGLang adds NVFP4 storage to its paged KV cache, cutting per-token key-value bytes to about 56% of FP8. On Qwen3.8-27B at 32K, 160K and 1M input lengths with matched decode concurrency, peak-batch decode throughput rises about 26–30%; on agentic load beyond concurrency 12, FP8 throughput collapses while NVFP4 keeps climbing. Accuracy gaps vary by task, reaching 1.60 points on SWE-bench Verified.

SGLang adds NVFP4 storage to its paged KV cache, cutting per-token key-value bytes to about 56% of FP8. On Qwen3.8-27B at 32K, 160K and 1M input lengths with matched decode concurrency, peak-batch decode throughput rises about 26–30%; on agentic load beyond concurrency 12, FP8 throughput collapses while NVFP4 keeps climbing. Accuracy gaps vary by task, reaching 1.60 points on SWE-bench Verified.

NVIDIA released Nemotron 3.5 Super VL, an open multimodal model now available for inference and training on Baseten, so agents can read documents, interpret application screens and analyze video directly. It builds on the Nemotron 3 Super architecture as a 120B-parameter Mixture-of-Experts model that activates 12B parameters per token, adding a vision encoder for image input. In NVIDIA's reported evaluations it scores an Elo of 1190 on GDPval versus 493 for Nemotron 3 Super, and also scores higher on SciCode and GPQA-diamond.

NVIDIA released Nemotron 3.5 Super VL, an open multimodal model now available for inference and training on Baseten, so agents can read documents, interpret application screens and analyze video directly. It builds on the Nemotron 3 Super architecture as a 120B-parameter Mixture-of-Experts model that activates 12B parameters per token, adding a vision encoder for image input. In NVIDIA's reported evaluations it scores an Elo of 1190 on GDPval versus 493 for Nemotron 3 Super, and also scores higher on SciCode and GPQA-diamond.

Sustained concurrency in asynchronous RLVR rollout concentrates context processing into fewer decode steps, making the KV cache rather than weights the dominant cost. KV4-RL stores the full context in calibration-free INT4 and protects only attention sinks and recent context in bounded INT8; across full-parameter GRPO training on Qwen3-4B-Base and Qwen3-8B-Base, its six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout while substantially accelerating rollout and full iterations.

Sustained concurrency in asynchronous RLVR rollout concentrates context processing into fewer decode steps, making the KV cache rather than weights the dominant cost. KV4-RL stores the full context in calibration-free INT4 and protects only attention sinks and recent context in bounded INT8; across full-parameter GRPO training on Qwen3-4B-Base and Qwen3-8B-Base, its six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout while substantially accelerating rollout and full iterations.

To address the KV cache memory bottleneck that grows with recurrent depth in looped Transformers, the authors propose ResidualQuant, which uses the final-loop KV states as an anchor and represents the remaining loops as low-precision residuals, combined with least-squares scaling, rotations applied to the residuals, and loop-wise INT4/INT2 mixed precision; across Ouro-1.4B and Huginn-3.5B on mathematical reasoning and code generation benchmarks it improves the accuracy-memory tradeoff over rotation-based KV quantization, reducing theoretical KV storage by 80.7% relative to BF16 while retaining accuracy close to BF16 under mixed precision.

To address the KV cache memory bottleneck that grows with recurrent depth in looped Transformers, the authors propose ResidualQuant, which uses the final-loop KV states as an anchor and represents the remaining loops as low-precision residuals, combined with least-squares scaling, rotations applied to the residuals, and loop-wise INT4/INT2 mixed precision; across Ouro-1.4B and Huginn-3.5B on mathematical reasoning and code generation benchmarks it improves the accuracy-memory tradeoff over rotation-based KV quantization, reducing theoretical KV storage by 80.7% relative to BF16 while retaining accuracy close to BF16 under mixed precision.