XM-Sparse moves sparse-attention KV cache off HBM: 49.25% lower time per token at 16K context and matched batch

Sparse attention reads only selected KV entries at each decode step but still retains the full KV cache because the selection changes during generation; XM-Sparse places that full cache in a high-bandwidth memory tier outside the GPU so decode GPUs stop fetching from host DRAM. SGLang HiSparse previously used a hot KV buffer to cut external-memory reads and kept the indexer key cache in GPU HBM, while external memory remained host DRAM over PCIe. At 16K context, with both configurations at the same batch sizes and the same amount of external memory, average time per output token at batch 192 per GPU drops from 167.48 ms to 84.99 ms at 20% misses and from 255.69 ms to 93.13 ms at 40% misses.

The approach treats X-Mem™ as a shared memory node in the rack: prefill GPUs write prompt KV directly into it, and decode GPUs fetch selected entries over the scale-up interconnect when the HBM hot buffer lacks them. An X-Mem™ memory processing unit node holds four MPUs with up to 24 TB of DDR5 and about 4 TB/s of aggregate bandwidth, exposed as GPU-addressable memory through CUDA- and PyTorch-compatible APIs. The test runs GLM-5.3 on four NVIDIA B300 decode GPUs with attention DP4 and MoE EP4, using four additional B300 GPUs whose HBM stands in for X-Mem™ memory behind the same access path the silicon will expose, with synthetic KV prefixes, fixed miss rates and prefetch disabled.

Capacity and bandwidth jointly set throughput: longer contexts make full KV caches larger, so 1 TB of external memory caps batch size first, and only after expanding to 24 TB does bandwidth become the new limit. At 128K context the 1 TB configuration is limited to batch 36 per GPU even with NVLink-C2C; expanding to 24 TB allows batch 109, but PCIe-CXL bandwidth then becomes the bottleneck. At 128K and batch 109, the 900 GB/s scale-up fabric delivers 1.39× the throughput of 64 GB/s PCIe-CXL; at 1M context, expanding capacity from 1 TB to 24 TB raises the feasible batch from 4 to 20 per GPU and gives 2.34× the throughput at the same bandwidth.

Results cover 16K, 128K and 1M contexts, fixed top-k miss rates of 20% and 40%, synthetic KV prefixes with prefetch disabled, and GLM-5.3 on four B300 decode GPUs; the X-Mem™ side is stood in for by four B300 GPUs' HBM, and capacity is modeled as a 24 TB budget rather than physical memory.

X-Mem™ silicon is not yet available, and the current data come from the X-Mem™ Pilot Virtual Machine; at small batches X-Mem™ differs little from host DRAM and can be slightly slower, with the benefit becoming pronounced only as batch grows.