SGLang stores KV cache in 4 bits: long-context decode throughput rises 26–30%, and FP8 stalls first under agent concurrency

Long-context inference can now keep its KV cache in 4-bit form and read the compressed keys and values directly during decoding, instead of decompressing the whole cache first. Production systems have generally stored KV in FP8, which already halves the data relative to BF16, but 4-bit formats carry larger quantization error, the main obstacle to compressing further. On one NVIDIA RTX PRO 6000 Blackwell GPU, Qwen3.8-27B at 32K, 160K and 1M inputs with exactly 1,024 output tokens per request showed about 26–30% higher peak-batch decode throughput at matched decode concurrency.

The implementation connects the NVFP4 cache to SGLang's three attention paths: initial prefill, chunked prefill or extend, and decode. Initial prefill attends directly on the current prompt's FP8 keys and values while quantizing the same data into the NVFP4 cache; the extend path dequantizes the cached prefix into one FP8 workspace shared across layers and computes it alongside the current chunk. The decode path dequantizes inside the attention kernel, avoiding a separate read-write round trip over the whole cache; each block of 16 values takes 8 bytes of packed data plus 1 byte of block scale, about 56% of FP8 storage for the same number of keys and values.

Under capacity-driven load, the smaller cache keeps more requests resident on the same GPU, and the throughput gain grows with it. At matched concurrency the NVFP4 advantage comes from reduced read traffic; once concurrency is raised, NVFP4 also holds more requests, a benefit FP8 does not have. Resident requests at 32K, 160K and 1M rose from 44, 10 and 1 to 70, 15 and 2, with peak-batch decode throughput up 37.37%, 57.75% and 78.46%; prefill mean time to first token rose only 0.20–0.40% at matched concurrency.

The demonstrated scope is the experimental NVFP4 KV cache in SGLang, with accuracy evaluated on Qwen3.5-397B-A17B and Qwen3.8-27B and performance measured on one RTX PRO 6000 Blackwell GPU and an 8-GPU RTX 6000D node; the 1M workload is a performance-only experiment using an explicit context-length override, while the model's native limit is 262,144 tokens.

Support is still experimental and covers only SM12x and SM100/SM103 plus GQA and sparse MLA models; the experiments did not use NVFP4's per-tensor FP32 global scale (set to 1.0), and calibration may narrow the accuracy gap with FP8. Two models and a small task set do not establish a general relationship between model size and quantization tolerance, and the decode throughput gains are not equivalent to end-to-end speedups.