# Long-Context Modeling

4 related articles

Subscribe to updates

arXiv Oct 7, 2026
WavePrune caps each RoPE channel at its first rotation period, raising HELMET long-context scores on four of five models and reaching 1.15x prefill speedup at 32K context
The authors propose WavePrune, a training-free method that restricts each RoPE channel to its first rotation period, removing position-aliasing "ghost" sub-diagonals; it raises HELMET scores on four of five long-context models (35.7 to 40.0 on Qwen3-8B), lowers validation loss at extrapolated lengths when pretraining from scratch, and, via FlashWavePrune, delivers 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context.
The authors propose WavePrune, a training-free method that restricts each RoPE channel to its first rotation period, removing position-aliasing "ghost" sub-diagonals; it raises HELMET scores on four of five long-context models (35.7 to 40.0 on Qwen3-8B), lowers validation loss at extrapolated lengths when pretraining from scratch, and, via FlashWavePrune, delivers 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context.
Guancheng Du（First author）
WavePrune FlashWavePrune Rotary Position Embedding Long-Context Modeling Sparse Attention
arXiv Oct 1, 2026
FRAC swaps exponential forgetting for power-law long memory in SSMs, beating Mamba and GDN on long-context 1.3B language modeling
The work introduces FRAC, a selective state space model architecture derived from fractional dynamics that approximates a heavy-tailed fractional kernel with a finite-state, log-spaced sum of exponential modes, replacing exponential forgetting with power-law long memory and improving long-context performance over SSM baselines such as Mamba2, GDN, and Mamba3 on synthetic long-tail and recall tasks, 1.3B-parameter language modeling, and DNA modeling, while staying competitive on short-context tasks.
The work introduces FRAC, a selective state space model architecture derived from fractional dynamics that approximates a heavy-tailed fractional kernel with a finite-state, log-spaced sum of exponential modes, replacing exponential forgetting with power-law long memory and improving long-context performance over SSM baselines such as Mamba2, GDN, and Mamba3 on synthetic long-tail and recall tasks, 1.3B-parameter language modeling, and DNA modeling, while staying competitive on short-context tasks.
Ivan Kobyzev（First author）
Frac Fractional State Space Transition State-Space Models Long-Context Modeling Language modeling
arXiv Sep 28, 2026
PISA cuts block-sparse attention selection to O(N log N) with pyramid Top-K, running 9.95x faster than BSA at 256K
Researchers from Shanghai Jiao Tong University and ByteDance Seed propose PISA, a block-sparse attention mechanism that uses coarse-to-fine pyramid Top-K selection with LogSumExp scoring and hardware-aware Triton kernels to reduce block-selection complexity from O(N²/C) to O(N log N); across 418M, 1.47B, and 2.67B scales it matches BSA, NSA, and HiLS on language modeling and commonsense reasoning, achieves the highest average accuracy among sparse methods on six containment tasks, and speeds up block selection over BSA by 2.86x, 5.31x, and 9.95x at 64K, 128K, and 256K.
Researchers from Shanghai Jiao Tong University and ByteDance Seed propose PISA, a block-sparse attention mechanism that uses coarse-to-fine pyramid Top-K selection with LogSumExp scoring and hardware-aware Triton kernels to reduce block-selection complexity from O(N²/C) to O(N log N); across 418M, 1.47B, and 2.67B scales it matches BSA, NSA, and HiLS on language modeling and commonsense reasoning, achieves the highest average accuracy among sparse methods on six containment tasks, and speeds up block selection over BSA by 2.86x, 5.31x, and 9.95x at 64K, 128K, and 256K.
ByteDance Seed Shanghai Innovation Institute Shanghai Jiao Tong University ByteDance (China)
Bohao Tang（First author）
PISA (Pyramid Sparse Attention)Block-Sparse Attention Pyramid Top-K Block Selection LogSumExp Block Scoring Long-Context Modeling
arXiv Sep 14, 2026
ResonatorLM: Replacing Attention with Causal Resonant Field Mixing for Efficient Long-Context Language Modeling
This work introduces ResonatorLM, which replaces self-attention with a causal resonant field mixer built from damped resonators, treating a token sequence as a driven latent field, running training and prefill through O(T log T) causal FFT convolution and decoding through a fixed-size recurrent state; in a matched setting near 6M parameters, WikiText-2 character-level test perplexity drops from 4.617 to 3.764 and accuracy rises from 55.32% to 61.31%, decode reaches a 6.47x speedup over an optimized attention block at 32K tokens, and a separate kernel-level benchmark reports 440.29x at 8K and 575.86x at 32K.
This work introduces ResonatorLM, which replaces self-attention with a causal resonant field mixer built from damped resonators, treating a token sequence as a driven latent field, running training and prefill through O(T log T) causal FFT convolution and decoding through a fixed-size recurrent state; in a matched setting near 6M parameters, WikiText-2 character-level test perplexity drops from 4.617 to 3.764 and accuracy rises from 55.32% to 61.31%, decode reaches a 6.47x speedup over an optimized attention block at 32K tokens, and a separate kernel-level benchmark reports 440.29x at 8K and 575.86x at 32K.
University of San Francisco
Archie Chaudhury（First author, Corresponding author）
ResonatorLM Causal Resonant Field Mixing Long-Context Modeling

You’re all caught up

WavePrune caps each RoPE channel at its first rotation period, raising HELMET long-context scores on four of five models and reaching 1.15x prefill speedup at 32K context

FRAC swaps exponential forgetting for power-law long memory in SSMs, beating Mamba and GDN on long-context 1.3B language modeling

PISA cuts block-sparse attention selection to O(N log N) with pyramid Top-K, running 9.95x faster than BSA at 256K

ResonatorLM: Replacing Attention with Causal Resonant Field Mixing for Efficient Long-Context Language Modeling