# KV Cache

Long-context inference makes the KV cache grow linearly with sequence length and often exceed GPU memory. VerKV keeps the full cache in CPU memory and only a Poisson sample of tokens, drawn from importance scores, on the GPU; at each decode step a certificate estimates the attention error and extra entries are fetched from CPU only when the bound exceeds tolerance. Across Qwen3-4B, Qwen3.5-9B and Ministral-3-8B on RULER and LOFT at 32K and 128K context, with 3.13% of the cache resident on GPU, the three retain 94.6%, 84.6% and 79.6% of dense quality, where Fast KVzip and KVzip at the same 3.13% retain only 1.9% to 11.9%.

Long-context inference makes the KV cache grow linearly with sequence length and often exceed GPU memory. VerKV keeps the full cache in CPU memory and only a Poisson sample of tokens, drawn from importance scores, on the GPU; at each decode step a certificate estimates the attention error and extra entries are fetched from CPU only when the bound exceeds tolerance. Across Qwen3-4B, Qwen3.5-9B and Ministral-3-8B on RULER and LOFT at 32K and 128K context, with 3.13% of the cache resident on GPU, the three retain 94.6%, 84.6% and 79.6% of dense quality, where Fast KVzip and KVzip at the same 3.13% retain only 1.9% to 11.9%.

Meta's open-weights mixture-of-experts model Muse Spark Open has 816B total parameters and about 42B active per token; serving it on TPU v7 previously took 2.9× the time needed just to stream its weights from HBM for one verify step. SGLang-TorchTPU now writes all 62 layers of a decode or verify step as one Pallas program, issuing weight, KV and cross-device transfers ahead of the compute that needs them. Per output token, a single request drops from 3.78 ms to 1.95 ms, two concurrent requests gain 1.85×, and the gain narrows as concurrency grows.

Meta's open-weights mixture-of-experts model Muse Spark Open has 816B total parameters and about 42B active per token; serving it on TPU v7 previously took 2.9× the time needed just to stream its weights from HBM for one verify step. SGLang-TorchTPU now writes all 62 layers of a decode or verify step as one Pallas program, issuing weight, KV and cross-device transfers ahead of the compute that needs them. Per output token, a single request drops from 3.78 ms to 1.95 ms, two concurrent requests gain 1.85×, and the gain narrows as concurrency grows.

NVIDIA reframes AI-factory memory from how much capacity a GPU has to how many useful tokens each byte produces, combining NVFP4 weight and KV-cache compression, wide expert parallelism for MoE weights, Dynamo KV-cache reuse, Vera CPU staging of context during tool calls, and CMX pod-level context storage. On SemiAnalysis AgentX, which replays recorded coding-agent sessions, Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user, with both at 288 GB per GPU.

NVIDIA reframes AI-factory memory from how much capacity a GPU has to how many useful tokens each byte produces, combining NVFP4 weight and KV-cache compression, wide expert parallelism for MoE weights, Dynamo KV-cache reuse, Vera CPU staging of context during tool calls, and CMX pod-level context storage. On SemiAnalysis AgentX, which replays recorded coding-agent sessions, Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user, with both at 288 GB per GPU.

For coding, math and computer-use agent workloads, Evolving Infra decouples compute allocation between prefill and decode from KV cache management: a node switches phase without a restart, and its KV stays readable from a shared pool. On up to 256 GB300 GPUs it cuts mean end-to-end latency and P95 time to first token by up to 4.1 and 14 relative to NVIDIA Dynamo, and a phase switch takes 6.2 seconds on average against 754 seconds for a Dynamo relaunch.

For coding, math and computer-use agent workloads, Evolving Infra decouples compute allocation between prefill and decode from KV cache management: a node switches phase without a restart, and its KV stays readable from a shared pool. On up to 256 GB300 GPUs it cuts mean end-to-end latency and P95 time to first token by up to 4.1 and 14 relative to NVIDIA Dynamo, and a phase switch takes 6.2 seconds on average against 754 seconds for a Dynamo relaunch.

Long-context decoding is bottlenecked by repeatedly reading a growing KV cache, and sparse attention reads only part of it, yet token-level selection has not reliably delivered both accuracy and lower latency. JSSA scores candidate tokens in a shared low-dimensional query–key subspace and pairs it with a gather-free indexed-attention kernel that keeps a distinct selected token set for each KV head. Across GQA, hybrid sliding-window/global attention and MLA models it stays closest to full-KV accuracy, and on a B200 at 128K context its decoding throughput reaches up to that of full-KV attention.

Long-context decoding is bottlenecked by repeatedly reading a growing KV cache, and sparse attention reads only part of it, yet token-level selection has not reliably delivered both accuracy and lower latency. JSSA scores candidate tokens in a shared low-dimensional query–key subspace and pairs it with a gather-free indexed-attention kernel that keeps a distinct selected token set for each KV head. Across GQA, hybrid sliding-window/global attention and MLA models it stays closest to full-KV accuracy, and on a B200 at 128K context its decoding throughput reaches up to that of full-KV attention.

Netpreme attaches its X-Mem™ memory tier to SGLang HiSparse so decode GPUs fetch missed KV entries from high-bandwidth external memory instead of host DRAM over PCIe. At 16K context, batch 192 per GPU and a 20% top-k miss rate, average time per output token falls from 167.48 ms to 84.99 ms, a 49.25% reduction; at a 40% miss rate the reduction is 63.58%. At 20% misses, peak decode throughput is 2.09–3.04× that of host DRAM across 16K to 1M contexts.

Netpreme attaches its X-Mem™ memory tier to SGLang HiSparse so decode GPUs fetch missed KV entries from high-bandwidth external memory instead of host DRAM over PCIe. At 16K context, batch 192 per GPU and a 20% top-k miss rate, average time per output token falls from 167.48 ms to 84.99 ms, a 49.25% reduction; at a 40% miss rate the reduction is 63.58%. At 20% misses, peak decode throughput is 2.09–3.04× that of host DRAM across 16K to 1M contexts.