# Linear Attention

9 related articles

Subscribe to updates

arXiv Oct 28, 2026
Nonlinear memory stops approximating by blocks: ParaTTT turns test-time training layers into parallel refinement, and its 340M model trains faster than TTT-MLP
Test-time training layers replace the recurrent state with a small neural network trained online by gradient descent, which is more expressive than a linear memory but forces sequential computation because each token's gradient depends on every earlier update. ParaTTT instead iteratively refines an estimate of the hidden-layer pre-activation gradients, with each refinement built from a forward, a readout and a backward chunkwise-parallel pass, and the exact online recurrence is a fixed point of that iteration. At 4K context, two refinements reach the error of TTT-MLP's block approximation in less time, and 340M and 1.3B models match or exceed Gated DeltaNet and Mamba-2 on perplexity and recall.
Test-time training layers replace the recurrent state with a small neural network trained online by gradient descent, which is more expressive than a linear memory but forces sequential computation because each token's gradient depends on every earlier update. ParaTTT instead iteratively refines an estimate of the hidden-layer pre-activation gradients, with each refinement built from a forward, a readout and a backward chunkwise-parallel pass, and the exact online recurrence is a fixed point of that iteration. At 4K context, two refinements reach the error of TTT-MLP's block approximation in less time, and 340M and 1.3B models match or exceed Gated DeltaNet and Mamba-2 on perplexity and recall.
Massachusetts Institute of Technology
Noel Loo（First author）
ParaTTT Test-Time Training Linear Attention Language modeling
arXiv Oct 12, 2026
Attention linearization stops treating all dimensions alike: grouping the few outliers cuts Qwen2.5-0.5B perplexity from 31.71 to 27.50
Approximation error in pretrained attention concentrates in a few high-contribution query–key dimensions, and sharing quadratic-kernel coefficients by contribution group lowers frozen-attention relative MSE to 0.432 on Qwen2.5-0.5B and WikiText-103 perplexity from 31.71 to 27.50 after replacing middle-layer attention.
National University of Singapore Beijing Normal University, Zhuhai
Weiqi Sun（First author）
Outlier-Aware Attention Linearization Linear Attention Softmax Attention Linearization Quadratic Kernel Attention Language modeling
NVIDIA Technical Blog Oct 20, 2026
Mamba-2's forward pass moves into Python kernels: 1.14x to 1.82x faster than Triton, with gains growing at longer sequences
Machine learning engineers at Jane Street, working with NVIDIA, rewrote the Mamba-2 forward pass in NVIDIA CUDA Tile (cuTile), a Python-based GPU programming model, and tuned it for Blackwell Tensor Cores. Mamba-2 combines a fixed-size recurrent state with attention-style matrix multiplication, but its forward pass still needs efficient kernels. On a single B300, the kernels run 1.14x to 1.82x faster than a Triton implementation, with speedups growing as sequence length and state size increase; these are kernel-level results, and end-to-end training also depends on the backward pass.
Machine learning engineers at Jane Street, working with NVIDIA, rewrote the Mamba-2 forward pass in NVIDIA CUDA Tile (cuTile), a Python-based GPU programming model, and tuned it for Blackwell Tensor Cores. Mamba-2 combines a fixed-size recurrent state with attention-style matrix multiplication, but its forward pass still needs efficient kernels. On a single B300, the kernels run 1.14x to 1.82x faster than a Triton implementation, with speedups growing as sequence length and state size increase; these are kernel-level results, and end-to-end training also depends on the backward pass.
NVIDIA
NVIDIA CUDA Tile (cuTile)Mamba-2 TileGym Linear Attention
arXiv Oct 12, 2026
3D reconstruction drops dense attention layer by layer: up to 5.6x end-to-end speedup at 1,024 views
Zebra3D swaps dense global attention in pretrained 3D reconstruction models for a per-layer mix of linear and tile-sparse attention, holding pose accuracy close to dense models at roughly 94% mean attention sparsity and reaching up to 5.6x end-to-end speedup at 1,024 views.
Advanced Micro Devices
Utkarsh Saxena（First author）
Zebra3D Linear Attention Sparse Attention 3D Reconstruction Knowledge Distillation
arXiv Oct 9, 2026
V-CoLA compresses vision tokens using linear-attention intrinsic signals, retaining 88% performance at 12.5% tokens with up to 6.15x prefill speedup
The authors propose V-CoLA, a training-free vision token compression framework for hybrid vision-language models with linear attention, deriving a uniqueness-aware importance criterion from the retention preferences of the bounded recurrent state, combined with adaptive chunk-wise merging and early exiting of deep-layer vision tokens; on Qwen3.5-9B and related models it reaches 99.5% of original performance at 50% vision tokens and over 88.0% at 12.5%, with 1.86 to 6.15 prefill speedup.
The authors propose V-CoLA, a training-free vision token compression framework for hybrid vision-language models with linear attention, deriving a uniqueness-aware importance criterion from the retention preferences of the bounded recurrent state, combined with adaptive chunk-wise merging and early exiting of deep-layer vision tokens; on Qwen3.5-9B and related models it reaches 99.5% of original performance at 50% vision tokens and over 88.0% at 12.5%, with 1.86 to 6.15 prefill speedup.
Alibaba Group (China)
Hao Jiang（First author）
V-CoLA Visual Token Compression Linear Attention Inference acceleration
arXiv Oct 9, 2026
Nonlinear attention makes loss decay inversely with depth, while linear attention stays bound to the data spectrum
In controlled in-context learning tasks comparing linear and nonlinear attention, the authors find that nonlinear attention can selectively focus on relevant tokens, letting strong and weak spectral directions be learned in parallel and producing inverse-depth loss decay, whereas linear attention's depth scaling depends on the data spectrum and quickly plateaus.
Peking University Tsinghua University Massachusetts Institute of Technology
Zirui Peng（First author）
Inverse-Depth Scaling Nonlinear Attention Linear Attention Neural Scaling Laws In-Context Learning
arXiv Oct 8, 2026
Universal Attention composes decay gates for over 10x KV-cache compression and beats unpruned Llama3 on RULER
The work introduces Universal Attention, which combines query-aware, key-aware, and similarity-based decay mechanisms via a geometric mean into an additive bias on the attention matrix, letting the model adaptively prune its KV-cache while keeping RoPE and Softmax; on a 1B Llama3-style model trained on 50B tokens it achieves over 10x compression, with RULER averages of 50.3/49.3 versus 40.8 for unpruned Llama3 at only 21.4/12.9 MB of cache.
The work introduces Universal Attention, which combines query-aware, key-aware, and similarity-based decay mechanisms via a geometric mean into an additive bias on the attention matrix, letting the model adaptively prune its KV-cache while keeping RoPE and Softmax; on a 1B Llama3-style model trained on 50B tokens it achieves over 10x compression, with RULER averages of 50.3/49.3 versus 40.8 for unpruned Llama3 at only 21.4/12.9 MB of cache.
University of Illinois Urbana-Champaign IBM (United States)
Davis Wertheimer（First author）
Universal Attention KV-cache pruning Self-Attention KV Cache Linear Attention
arXiv Oct 2, 2026
LOCI conditions a linear memory on camera geometry alongside historical KV, reaching 14.36 dB revisit PSNR on the MIND memory benchmark and streaming 300 seconds at a constant 23.6 GiB
LOCI introduces a hybrid spatial-memory architecture in which 15 of 30 transformer blocks keep a key-value cache of past observations while the other 15 are restricted to the current chunk and complemented by a recurrent linear-attention memory whose reads and writes are conditioned on projective camera geometry, with recurrent readouts flowing into subsequent cache-backed blocks to supply their queries with accumulated scene context; on the public MIND memory benchmark and on held-out recorded trajectories it reproduces revisited content more faithfully than representative world models and a same-recipe full-softmax model, lowers peak memory at equal length by about 30% with full history, and streams long videos at constant memory with a bounded bank of retained observations.
LOCI introduces a hybrid spatial-memory architecture in which 15 of 30 transformer blocks keep a key-value cache of past observations while the other 15 are restricted to the current chunk and complemented by a recurrent linear-attention memory whose reads and writes are conditioned on projective camera geometry, with recurrent readouts flowing into subsequent cache-backed blocks to supply their queries with accumulated scene context; on the public MIND memory benchmark and on held-out recorded trajectories it reproduces revisited content more faithfully than representative world models and a same-recipe full-softmax model, lowers peak memory at equal length by about 30% with full history, and streams long videos at constant memory with a bounded bank of retained observations.
Ji Xia（First author）
LOCI Video World Model Kimi Delta Attention Revisit Consistency Linear Attention
arXiv Oct 1, 2026
A survey recasts attention evolution as contextual-memory organization, using 59 release records and 11 open-weight endpoints
Treating model-internal contextual memory as the shared unit of analysis, this survey introduces five dimensions—Memory Representation, Memory Update, Access, Readout, and Integration—and reviews five research lines (Softmax Attention, Sparse Attention, Linear Attention, State Space Models, and Hybrid Architectures), using 59 release-level records across 14 major model lineages and a frozen comparison of 11 high-performing open-weight endpoints through September 22, 2026 to argue that attention design is diversifying rather than converging, with hybridization and cross-layer reuse making network depth a dimension along which contextual memory is organized.
Treating model-internal contextual memory as the shared unit of analysis, this survey introduces five dimensions—Memory Representation, Memory Update, Access, Readout, and Integration—and reviews five research lines (Softmax Attention, Sparse Attention, Linear Attention, State Space Models, and Hybrid Architectures), using 59 release-level records across 14 major model lineages and a frozen comparison of 11 high-performing open-weight endpoints through September 22, 2026 to argue that attention design is diversifying rather than converging, with hybridization and cross-layer reuse making network depth a dimension along which contextual memory is organized.
Zhentao Tan（First author）
Contextual Memory Attention Mechanism Hybrid Architecture Linear Attention Sparse Attention

You’re all caught up