# Hybrid LLM Serving

Hybrid language models compress history into fixed-size recurrent states, yet serving systems still treat those states as dense matrices, reading and rewriting them whole at every decoding step, so memory traffic grows with batch size. SinkState exploits the concentration of queries, keys and values in linear attention into a stable low-dimensional subspace, keeping each state in an adaptive low-rank factorization throughout decoding, merging new tokens with a local SVD and consolidating the frozen part asynchronously on the CPU. Across four Mamba-2 and GDN hybrid models from 4B to 120B parameters, peak recurrent-state memory falls to 0.38–0.63 of the baseline and decoding speeds up by up to 1.58x, with accuracy maintained.

Hybrid language models compress history into fixed-size recurrent states, yet serving systems still treat those states as dense matrices, reading and rewriting them whole at every decoding step, so memory traffic grows with batch size. SinkState exploits the concentration of queries, keys and values in linear attention into a stable low-dimensional subspace, keeping each state in an adaptive low-rank factorization throughout decoding, merging new tokens with a local SVD and consolidating the frozen part asynchronously on the CPU. Across four Mamba-2 and GDN hybrid models from 4B to 120B parameters, peak recurrent-state memory falls to 0.38–0.63 of the baseline and decoding speeds up by up to 1.58x, with accuracy maintained.