SinkState stores recurrent states as adaptive low-rank factors: hybrid-model decoding peak memory falls to as little as 1/2.6 of baseline
Synopsis
Hybrid language models compress history into fixed-size recurrent states, yet serving systems still treat those states as dense matrices, reading and rewriting them whole at every decoding step, so memory traffic grows with batch size. SinkState exploits the concentration of queries, keys and values in linear attention into a stable low-dimensional subspace, keeping each state in an adaptive low-rank factorization throughout decoding, merging new tokens with a local SVD and consolidating the frozen part asynchronously on the CPU. Across four Mamba-2 and GDN hybrid models from 4B to 120B parameters, peak recurrent-state memory falls to 0.38–0.63 of the baseline and decoding speeds up by up to 1.58x, with accuracy maintained.
Interpretation
During decoding, a hybrid model's recurrent state is no longer read and written as a dense matrix; it is stored as a low-rank factorization whose rank differs across states and changes as decoding proceeds. Earlier serving systems reduced this overhead through state reuse or deferred state writes but kept dense representations, while state pruning and calibrated projections relied on static compression choices. On four hybrid models, Nemotron-3-Nano-4B, Nemotron-3-Super-120B-A12B, Qwen3.5-9B and Qwen3-Next-80B-A3B-Instruct, peak recurrent-state memory falls to 0.38–0.63 of the baseline, up to 2.6x smaller and 1.9x on average.
Incoming keys and values are mostly absorbed by the leading components of the state, so each update needs only a small SVD on the top subspace while the remaining components stay frozen. Recomputing an exact SVD of every state after each update is prohibitive: at batch size 128, running it on all states of Nemotron-3-Nano-4B takes about 4000 ms, far longer than a typical 20–30 ms decoding step. SinkState buffers new tokens' keys and values and merges them in batches, staggering compaction across layers; on Nemo-4B, staggering lowers P99 step time by 40.9 ms.
The frozen part accumulates redundant components with similar directions, so SinkState periodically consolidates it with an exact SVD on the CPU while GPU decoding continues. With local SVD alone, the frozen part stores noticeably more components than an exact SVD requires, offsetting the savings from low-rank storage. Adding CPU consolidation multiplies state memory by a further 0.86 and 0.89 on the two Nemotron models and by 0.62 and 0.65 on the two Qwen models.
Perspective
The evaluation covers four hybrid models: Nemotron-3-Nano-4B and Nemotron-3-Super-120B-A12B with Mamba-2 recurrent layers, and Qwen3.5-9B and Qwen3-Next-80B-A3B-Instruct with GDN layers; all experiments run on a single NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB of memory, with recurrent states kept in FP32.
Speedups vary by model and benchmark: 1.30–1.58x on Nemo-4B but only 1.04–1.16x on Nemo-120B, where MoE layers take a larger share of each step and the larger weights leave less memory for large batches; on Qwen models the speedup is also lower on GPQA Diamond and LiveBench, whose long outputs keep batches small. Accuracy changes show no consistent direction, improving in 12 of 24 model-benchmark pairs; Nemo-4B's 4.1-point gain on GPQA Diamond arises because the baseline reaches the generation limit on more questions.