Ze-Wei Liou

State-space model decoding previously read and wrote the full recurrent state at every step, making state traffic the memory-bandwidth bottleneck and leaving speculative decoding without cheap rollback. ReplaySSM represents the state as a checkpoint plus a buffer of recent inputs, writes the checkpoint only when the buffer fills, and computes outputs directly on most steps. Implemented in vLLM for Mamba-2 and Gated DeltaNet, it speeds up standard decoding over a tuned baseline at serving batch size 256, reaches about the throughput of vLLM's speculative decoding on two 120B-class models, and separate state precisions reduce output KL divergence and downstream accuracy losses.

State-space model decoding previously read and wrote the full recurrent state at every step, making state traffic the memory-bandwidth bottleneck and leaving speculative decoding without cheap rollback. ReplaySSM represents the state as a checkpoint plus a buffer of recent inputs, writes the checkpoint only when the buffer fills, and computes outputs directly on most steps. Implemented in vLLM for Mamba-2 and Gated DeltaNet, it speeds up standard decoding over a tuned baseline at serving batch size 256, reaches about the throughput of vLLM's speculative decoding on two 120B-class models, and separate state precisions reduce output KL divergence and downstream accuracy losses.