# Token Selection

Long-context decoding is bottlenecked by repeatedly reading a growing KV cache, and sparse attention reads only part of it, yet token-level selection has not reliably delivered both accuracy and lower latency. JSSA scores candidate tokens in a shared low-dimensional query–key subspace and pairs it with a gather-free indexed-attention kernel that keeps a distinct selected token set for each KV head. Across GQA, hybrid sliding-window/global attention and MLA models it stays closest to full-KV accuracy, and on a B200 at 128K context its decoding throughput reaches up to that of full-KV attention.

Long-context decoding is bottlenecked by repeatedly reading a growing KV cache, and sparse attention reads only part of it, yet token-level selection has not reliably delivered both accuracy and lower latency. JSSA scores candidate tokens in a shared low-dimensional query–key subspace and pairs it with a gather-free indexed-attention kernel that keeps a distinct selected token set for each KV head. Across GQA, hybrid sliding-window/global attention and MLA models it stays closest to full-KV accuracy, and on a B200 at 128K context its decoding throughput reaches up to that of full-KV attention.

Recurrent-memory models compress history into a fixed set of slots, and earlier multi-slot memories wrote every token to the same number of slots, merging whether to store a token with how many slots to update. Factorized Write Routing separates the two: a learned gate first decides whether the current token is written, then a slot router chooses a fixed or token-adaptive number of slots, while queries still read all slots. Across ten seeds on two synthetic long-context tasks with four slots per write, the result reverses with information density: always writing reaches 0.472 long-context accuracy on dense Feature Recall versus 0.337 for the learned gate, while the learned gate reaches 0.222 on sparse-fact Passkey versus 0.048 for always writing.

Recurrent-memory models compress history into a fixed set of slots, and earlier multi-slot memories wrote every token to the same number of slots, merging whether to store a token with how many slots to update. Factorized Write Routing separates the two: a learned gate first decides whether the current token is written, then a slot router chooses a fixed or token-adaptive number of slots, while queries still read all slots. Across ten seeds on two synthetic long-context tasks with four slots per write, the result reverses with information density: always writing reaches 0.472 long-context accuracy on dense Feature Recall versus 0.337 for the learned gate, while the learned gate reaches 0.222 on sparse-fact Passkey versus 0.048 for always writing.

Without retraining, this work retains only the tokens with the highest attention weights at each head, layer, and query while keeping their original weights unchanged, and estimates the effective attention set size needed to stay within a chosen loss tolerance by measuring the increase in negative log-likelihood (NLL); it finds that relatively small selected sets keep NLL close to the full-attention baseline, that attention-based selection substantially outperforms random selection, that the required set size grows with context while its fraction of context decreases, that in BABILong experiments with a fixed annotated supporting fact additional background pushes support tokens down the attention ranking and reduces their attention mass, and that renormalizing the retained weights can subs

Without retraining, this work retains only the tokens with the highest attention weights at each head, layer, and query while keeping their original weights unchanged, and estimates the effective attention set size needed to stay within a chosen loss tolerance by measuring the increase in negative log-likelihood (NLL); it finds that relatively small selected sets keep NLL close to the full-attention baseline, that attention-based selection substantially outperforms random selection, that the required set size grows with context while its fraction of context decreases, that in BABILong experiments with a fixed annotated supporting fact additional background pushes support tokens down the attention ranking and reduces their attention mass, and that renormalizing the retained weights can subs

The work reframes token selection in on-policy distillation (OPD) as a gradient-estimation reliability problem at a fixed prefix, decomposes the one-sample reverse-KL gradient into signal and sampling noise in information geometry, proposes the information-efficiency ratio (IER) as a signal-to-noise ratio under the optimal scalar baseline, and approximates it on a top-K candidate set for ranking; on mathematical and medical reasoning, IER alone approaches or exceeds full OPD at token budgets of 0.1%–1%, and combined with existing usefulness scores (IER-OR, IER-AND) it matches or exceeds full OPD in multiple settings.

The work reframes token selection in on-policy distillation (OPD) as a gradient-estimation reliability problem at a fixed prefix, decomposes the one-sample reverse-KL gradient into signal and sampling noise in information geometry, proposes the information-efficiency ratio (IER) as a signal-to-noise ratio under the optimal scalar baseline, and approximates it on a top-K candidate set for ranking; on mathematical and medical reasoning, IER alone approaches or exceeds full OPD at token budgets of 0.1%–1%, and combined with existing usefulness scores (IER-OR, IER-AND) it matches or exceeds full OPD in multiple settings.