# Mamba-2

Machine learning engineers at Jane Street, working with NVIDIA, rewrote the Mamba-2 forward pass in NVIDIA CUDA Tile (cuTile), a Python-based GPU programming model, and tuned it for Blackwell Tensor Cores. Mamba-2 combines a fixed-size recurrent state with attention-style matrix multiplication, but its forward pass still needs efficient kernels. On a single B300, the kernels run 1.14x to 1.82x faster than a Triton implementation, with speedups growing as sequence length and state size increase; these are kernel-level results, and end-to-end training also depends on the backward pass.

Machine learning engineers at Jane Street, working with NVIDIA, rewrote the Mamba-2 forward pass in NVIDIA CUDA Tile (cuTile), a Python-based GPU programming model, and tuned it for Blackwell Tensor Cores. Mamba-2 combines a fixed-size recurrent state with attention-style matrix multiplication, but its forward pass still needs efficient kernels. On a single B300, the kernels run 1.14x to 1.82x faster than a Triton implementation, with speedups growing as sequence length and state size increase; these are kernel-level results, and end-to-end training also depends on the backward pass.

State-space model decoding previously read and wrote the full recurrent state at every step, making state traffic the memory-bandwidth bottleneck and leaving speculative decoding without cheap rollback. ReplaySSM represents the state as a checkpoint plus a buffer of recent inputs, writes the checkpoint only when the buffer fills, and computes outputs directly on most steps. Implemented in vLLM for Mamba-2 and Gated DeltaNet, it speeds up standard decoding over a tuned baseline at serving batch size 256, reaches about the throughput of vLLM's speculative decoding on two 120B-class models, and separate state precisions reduce output KL divergence and downstream accuracy losses.

State-space model decoding previously read and wrote the full recurrent state at every step, making state traffic the memory-bandwidth bottleneck and leaving speculative decoding without cheap rollback. ReplaySSM represents the state as a checkpoint plus a buffer of recent inputs, writes the checkpoint only when the buffer fills, and computes outputs directly on most steps. Implemented in vLLM for Mamba-2 and Gated DeltaNet, it speeds up standard decoding over a tuned baseline at serving batch size 256, reaches about the throughput of vLLM's speculative decoding on two 120B-class models, and separate state precisions reduce output KL divergence and downstream accuracy losses.

Hybrid models interleave state-space layers with a few attention layers, and the state-space layers use a fixed-size state instead of a key-value cache that grows with context, yet at serving batch sizes they become the latency bottleneck and complicate speculative decoding and quantization. ReplaySSM represents the state as a checkpoint plus a buffer of recent inputs, separating output computation from state updates so most steps append inputs without writing state back. Across four hybrid models, the kernel runs 1.46x to 1.80x faster and the decode step 1.20x to 1.47x faster at batch size 256, and two 120B-class models reach 1.85x and 2.10x the speculative decoding throughput of vLLM at batch size 512.

Hybrid models interleave state-space layers with a few attention layers, and the state-space layers use a fixed-size state instead of a key-value cache that grows with context, yet at serving batch sizes they become the latency bottleneck and complicate speculative decoding and quantization. ReplaySSM represents the state as a checkpoint plus a buffer of recent inputs, separating output computation from state updates so most steps append inputs without writing state back. Across four hybrid models, the kernel runs 1.46x to 1.80x faster and the decode step 1.20x to 1.47x faster at batch size 256, and two 120B-class models reach 1.85x and 2.10x the speculative decoding throughput of vLLM at batch size 512.