WavePrune caps each RoPE channel at its first rotation period, raising HELMET long-context scores on four of five models and reaching 1.15x prefill speedup at 32K context

WavePrune truncates attention in each RoPE channel to a local window whose size equals one rotation wavelength, yielding a fine-grained mask over both sequence and head dimensions rather than a token-level sliding window. Prior work read slash structures in attention maps as useful long-context machinery; this work attributes part of them to position aliasing from RoPE periodicity and proves that, with identical queries and keys, the largest logit is approximately replicated onto infinitely many lags. A theorem with a direct proof, plus a stronger version in the appendix allowing random symmetric query and key distributions; empirically, attention maps averaged over 100 PG19 sequences of 4096 tokens for Qwen3-8B and Ministral3-3B, with per-channel logit decomposition.

Applied training-free, WavePrune raises the HELMET average on four of five long-context models, with Qwen3-8B moving from 35.7 to 40.0 and Recall and ICL improving by 6.0 and 10.0 points. The change needs no fine-tuning and leaves the RoPE frequency spectrum untouched, so it can be combined with frequency-rescaling methods such as YaRN. HELMET subtask means with standard deviations are reported for Qwen3-8B, Qwen3-32B, Gemma3-12B, Ministral3-3B and Llama3.1-8B, alongside general benchmarks run through lm-evaluation-harness; Llama3.1-8B degrades and is diagnosed in an appendix.

When pretraining from scratch, enabling WavePrune consistently lowers validation loss at extrapolated lengths under a 2048-token training and 4096-token validation setup, and outperforms enabling it only at inference. This indicates WavePrune is not merely an inference-time attention correction but changes the length-generalizable patterns the model learns. Paired runs at GPT2-base (117M), GPT2-medium (345M) and GPT2-large (774M) using nanoGPT on OpenWebText; extrapolated loss 3.0953 vs. 3.3314 for GPT2-medium and 2.9536 vs. 3.1797 for GPT2-large, with in-distribution validation loss comparable to baseline.

FlashWavePrune groups head dimensions into blocks of 16 sharing one threshold, skips fully masked groups and bypasses their key-cache loads, reaching 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. It turns RoPE-frequency-dependent channel sparsity into GEMM groups executable on Tensor Cores, unifying pruning along the sequence and head dimensions. Measured on an NVIDIA A100 in bf16 with batch size 32, GQA with 32 query and 8 KV heads, head dimension 128, reporting medians of 10 prefill and 50 decode iterations; the backward kernel overhead is 8.58%, 3.81% and 0.18% at 8K, 16K and 32K.

The result targets RoPE-based Transformer language models and applies to both long-context evaluation and from-scratch pretraining. Training-free integration works on Qwen3-8B, Qwen3-32B, Gemma3-12B and Ministral3-3B, and the authors state that WavePrune derives each channel's wavelength from the model's updated frequencies (e.g., YaRN), so it can be layered on frequency-rescaling methods. Efficiency gains target GPUs with Tensor Cores, giving 1.15x prefill and 1.24x decoding at head dimension 128 and 32K context, widening with sequence length; the authors also note future KV-cache memory savings by evicting cached key components once their distance from the current query exceeds the thresholds. Pretraining gains are measured at GPT2-base/medium/large with training length 2048 and validation length 4096, and the authors report the advantage persists when the RoPE base is increased after training.

Training-free gains are not uniform across models: the authors observe a drop on Llama3.1-8B and attribute it in an appendix to heads that rely on a high-frequency channel with wavelength of roughly a few hundred to stably transmit retrieval signals far beyond that wavelength; the remedies they tried did not recover the unpruned baseline, so which models can take the change directly remains open. The authors also note that pruning thresholds are currently shared across all attention heads although heads may play different roles, leaving per-head or learned thresholds unexplored, and that the fine-grained mask depends on RoPE, so generalizing it to other positional encodings and architectures is future work. The theory omits rotation frequencies when defining channel sensitivity, and the authors state the mathematically defined truncation sparsity may not always be small, leaving tighter analysis open. In addition, the HELMET evaluation replaces LLM-as-a-Judge with ROUGE-1 F1 for NarrativeQA and drops the Summ subtask, so specific numbers should be read together with the appendix tables.