# Sparse Attention

Query-conditioned sparse KV selection must wait for the target-layer query before scoring candidates and generating indices, so routing sits on the immediate pre-attention path. EarlySelectKV replaces the routing query with a proxy built from an earlier hidden state, precomputing the KV indices the next layer will access while final attention still uses the real query. On RocketKV it differs by +0.018 points on LongBench, about −0.01 points on RULER and +0.25 percentage points on representative NIAH/Passkey settings; in a TensorRT-LLM deployment with CPU-offloaded routing, end-to-end output throughput rises by up to 12.4% and GPU-resident memory falls by 11.1% of the combined KV/K-summary pool.

Query-conditioned sparse KV selection must wait for the target-layer query before scoring candidates and generating indices, so routing sits on the immediate pre-attention path. EarlySelectKV replaces the routing query with a proxy built from an earlier hidden state, precomputing the KV indices the next layer will access while final attention still uses the real query. On RocketKV it differs by +0.018 points on LongBench, about −0.01 points on RULER and +0.25 percentage points on representative NIAH/Passkey settings; in a TensorRT-LLM deployment with CPU-offloaded routing, end-to-end output throughput rises by up to 12.4% and GPU-resident memory falls by 11.1% of the combined KV/K-summary pool.

vLLM added same-day inference support for Mistral Large 4, a 1-trillion-parameter sparse mixture-of-experts multimodal model with 49 billion parameters active per token. The work spans breakable CUDA graphs, hand-written fusions, decode and prefill context parallelism, DSpark speculative decoding, NIXL disaggregated serving and host-memory KV offload. On TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 and holds about 1.5x the KV cache, while staying within 2.1 points of FP8 on multimodal and long-context evaluations.

vLLM added same-day inference support for Mistral Large 4, a 1-trillion-parameter sparse mixture-of-experts multimodal model with 49 billion parameters active per token. The work spans breakable CUDA graphs, hand-written fusions, decode and prefill context parallelism, DSpark speculative decoding, NIXL disaggregated serving and host-memory KV offload. On TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 and holds about 1.5x the KV cache, while staying within 2.1 points of FP8 on multimodal and long-context evaluations.

Long-context RLVR rollout generation now drafts tokens with sparse attention and verifies them with full attention, raising end-to-end training throughput to 1.39x and 1.35x of autoregressive on Qwen3-1.7B and Qwen3-8B while keeping comparable task performance.

Long-context RLVR rollout generation now drafts tokens with sparse attention and verifies them with full attention, raising end-to-end training throughput to 1.39x and 1.35x of autoregressive on Qwen3-1.7B and Qwen3-8B while keeping comparable task performance.

Zebra3D swaps dense global attention in pretrained 3D reconstruction models for a per-layer mix of linear and tile-sparse attention, holding pose accuracy close to dense models at roughly 94% mean attention sparsity and reaching up to 5.6x end-to-end speedup at 1,024 views.

Zebra3D swaps dense global attention in pretrained 3D reconstruction models for a per-layer mix of linear and tile-sparse attention, holding pose accuracy close to dense models at roughly 94% mean attention sparsity and reaching up to 5.6x end-to-end speedup at 1,024 views.

The authors argue that existing sparse-attention selectors keep large or high-mass entries of the attention matrix even though Transformers use that matrix through multiplication with value vectors, and propose MASA, which replaces attention-mass ranking with a closed-form score measuring each sparse unit's reduction in matrix-product approximation error; plugged into MInference, SeerAttention, and FlexPrefill without changing their sparse kernels, candidate units, or budgets, it yields consistent accuracy gains on RULER, InfiniteBench, and LongBench with negligible latency overhead.

The authors argue that existing sparse-attention selectors keep large or high-mass entries of the attention matrix even though Transformers use that matrix through multiplication with value vectors, and propose MASA, which replaces attention-mass ranking with a closed-form score measuring each sparse unit's reduction in matrix-product approximation error; plugged into MInference, SeerAttention, and FlexPrefill without changing their sparse kernels, candidate units, or budgets, it yields consistent accuracy gains on RULER, InfiniteBench, and LongBench with negligible latency overhead.

The authors propose WavePrune, a training-free method that restricts each RoPE channel to its first rotation period, removing position-aliasing "ghost" sub-diagonals; it raises HELMET scores on four of five long-context models (35.7 to 40.0 on Qwen3-8B), lowers validation loss at extrapolated lengths when pretraining from scratch, and, via FlashWavePrune, delivers 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context.

The authors propose WavePrune, a training-free method that restricts each RoPE channel to its first rotation period, removing position-aliasing "ghost" sub-diagonals; it raises HELMET scores on four of five long-context models (35.7 to 40.0 on Qwen3-8B), lowers validation loss at extrapolated lengths when pretraining from scratch, and, via FlashWavePrune, delivers 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context.