# Long-context inference

5 related articles

Subscribe to updates

arXiv Oct 14, 2026
JSSA makes token-level sparse decoding gather-free, reaching up to full-KV throughput at 128K context
Long-context decoding is bottlenecked by repeatedly reading a growing KV cache, and sparse attention reads only part of it, yet token-level selection has not reliably delivered both accuracy and lower latency. JSSA scores candidate tokens in a shared low-dimensional query–key subspace and pairs it with a gather-free indexed-attention kernel that keeps a distinct selected token set for each KV head. Across GQA, hybrid sliding-window/global attention and MLA models it stays closest to full-KV accuracy, and on a B200 at 128K context its decoding throughput reaches up to that of full-KV attention.
Long-context decoding is bottlenecked by repeatedly reading a growing KV cache, and sparse attention reads only part of it, yet token-level selection has not reliably delivered both accuracy and lower latency. JSSA scores candidate tokens in a shared low-dimensional query–key subspace and pairs it with a gather-free indexed-attention kernel that keeps a distinct selected token set for each KV head. Across GQA, hybrid sliding-window/global attention and MLA models it stays closest to full-KV accuracy, and on a B200 at 128K context its decoding throughput reaches up to that of full-KV attention.
Seoul National University
Jiwon Song（First author）
Joint Subspace Sparse Attention Sparse Attention KV Cache Token Selection Long-context inference
arXiv Oct 14, 2026
Titan-Llama bolts external neural memory onto a frozen Llama: long-document QA accuracy rises from 23.87% to 40.51%
Titan-Llama converts the first four layers of Llama-3.2-1B-Instruct to blockwise attention and attaches a fixed-size neural memory module to each KV head, using test-time training to compress historical key-value bindings while the backbone stays frozen. Against segmented attention windows alone, accuracy rises from 23.87% to 40.51% on QuALITY and from 14.50% to 40.75% on LongHealth. In the evaluated configurations, prefill is more than 20% faster, decode 37% faster at medium context lengths, and steady-state decode memory 24% lower.
Titan-Llama converts the first four layers of Llama-3.2-1B-Instruct to blockwise attention and attaches a fixed-size neural memory module to each KV head, using test-time training to compress historical key-value bindings while the backbone stays frozen. Against segmented attention windows alone, accuracy rises from 23.87% to 40.51% on QuALITY and from 14.50% to 40.75% on LongHealth. In the evaluated configurations, prefill is more than 20% faster, decode 37% faster at medium context lengths, and steady-state decode memory 24% lower.
Massachusetts Institute of Technology Cornell University
Sarah Pan（First author）
Titan-Llama Test-Time Training Neural Memory Module Long-context inference
Conference on Empirical Methods in Natural Language Processing (EMNLP) Nov 1, 2026
KV selection stops waiting for the target-layer query: EarlySelectKV precomputes indices from proxy queries, lifting long-output throughput by up to 12.4%
Query-conditioned sparse KV selection must wait for the target-layer query before scoring candidates and generating indices, so routing sits on the immediate pre-attention path. EarlySelectKV replaces the routing query with a proxy built from an earlier hidden state, precomputing the KV indices the next layer will access while final attention still uses the real query. On RocketKV it differs by +0.018 points on LongBench, about −0.01 points on RULER and +0.25 percentage points on representative NIAH/Passkey settings; in a TensorRT-LLM deployment with CPU-offloaded routing, end-to-end output throughput rises by up to 12.4% and GPU-resident memory falls by 11.1% of the combined KV/K-summary pool.
Query-conditioned sparse KV selection must wait for the target-layer query before scoring candidates and generating indices, so routing sits on the immediate pre-attention path. EarlySelectKV replaces the routing query with a proxy built from an earlier hidden state, precomputing the KV indices the next layer will access while final attention still uses the real query. On RocketKV it differs by +0.018 points on LongBench, about −0.01 points on RULER and +0.25 percentage points on representative NIAH/Passkey settings; in a TensorRT-LLM deployment with CPU-offloaded routing, end-to-end output throughput rises by up to 12.4% and GPU-resident memory falls by 11.1% of the combined KV/K-summary pool.
Chinese Academy of Sciences
Yuhao Liu（First author）Haifang Jian（Corresponding author）
EarlySelectKV KV Cache Sparse Attention Long-context inference
arXiv Oct 9, 2026
MASA recasts sparse-attention selection as matrix approximation, lifting long-context accuracy across three sparse frameworks at no extra budget
The authors argue that existing sparse-attention selectors keep large or high-mass entries of the attention matrix even though Transformers use that matrix through multiplication with value vectors, and propose MASA, which replaces attention-mass ranking with a closed-form score measuring each sparse unit's reduction in matrix-product approximation error; plugged into MInference, SeerAttention, and FlexPrefill without changing their sparse kernels, candidate units, or budgets, it yields consistent accuracy gains on RULER, InfiniteBench, and LongBench with negligible latency overhead.
The authors argue that existing sparse-attention selectors keep large or high-mass entries of the attention matrix even though Transformers use that matrix through multiplication with value vectors, and propose MASA, which replaces attention-mass ranking with a closed-form score measuring each sparse unit's reduction in matrix-product approximation error; plugged into MInference, SeerAttention, and FlexPrefill without changing their sparse kernels, candidate units, or budgets, it yields consistent accuracy gains on RULER, InfiniteBench, and LongBench with negligible latency overhead.
Duke University Stony Brook University
Fang Wan（First author）
Matrix Approximation Sparse Attention Sparse Attention Inference acceleration Long-context inference
arXiv Sep 29, 2026
DISCO splits long context into parallel Worker grounding plus Driver reasoning, holding 78.4% on 1M-token RULER-QA while cutting inference cost by over 80%
The work proposes DISCO, which separates long-context processing into local evidence grounding executed in parallel by lightweight Worker LLMs and global reasoning handled by a central Driver LLM whose dynamic DAG planning is optimized with GRPO reinforcement learning; with Qwen3-8B it holds 78.4% accuracy on RULER-QA at 1M tokens (where the RAG baseline collapses to 10.9%), improves up to 9.8 points over full-context baselines on LongBench v2, and matches Gemini-3-Pro-Preview while reducing inference cost by over 80%.
The work proposes DISCO, which separates long-context processing into local evidence grounding executed in parallel by lightweight Worker LLMs and global reasoning handled by a central Driver LLM whose dynamic DAG planning is optimized with GRPO reinforcement learning; with Qwen3-8B it holds 78.4% accuracy on RULER-QA at 1M tokens (where the RAG baseline collapses to 10.9%), improves up to 9.8 points over full-context baselines on LongBench v2, and matches Gemini-3-Pro-Preview while reducing inference cost by over 80%.
National University of Singapore Adobe Systems (United States)
Guanzheng Chen（First author）
DISCO Grounding-Reasoning Disaggregation GRPO Long-context inference

You’re all caught up

JSSA makes token-level sparse decoding gather-free, reaching up to full-KV throughput at 128K context

Titan-Llama bolts external neural memory onto a frozen Llama: long-document QA accuracy rises from 23.87% to 40.51%

KV selection stops waiting for the target-layer query: EarlySelectKV precomputes indices from proxy queries, lifting long-output throughput by up to 12.4%

MASA recasts sparse-attention selection as matrix approximation, lifting long-context accuracy across three sparse frameworks at no extra budget

DISCO splits long context into parallel Worker grounding plus Driver reasoning, holding 78.4% on 1M-token RULER-QA while cutting inference cost by over 80%