# Language modeling

Test-time training layers replace the recurrent state with a small neural network trained online by gradient descent, which is more expressive than a linear memory but forces sequential computation because each token's gradient depends on every earlier update. ParaTTT instead iteratively refines an estimate of the hidden-layer pre-activation gradients, with each refinement built from a forward, a readout and a backward chunkwise-parallel pass, and the exact online recurrence is a fixed point of that iteration. At 4K context, two refinements reach the error of TTT-MLP's block approximation in less time, and 340M and 1.3B models match or exceed Gated DeltaNet and Mamba-2 on perplexity and recall.

Test-time training layers replace the recurrent state with a small neural network trained online by gradient descent, which is more expressive than a linear memory but forces sequential computation because each token's gradient depends on every earlier update. ParaTTT instead iteratively refines an estimate of the hidden-layer pre-activation gradients, with each refinement built from a forward, a readout and a backward chunkwise-parallel pass, and the exact online recurrence is a fixed point of that iteration. At 4K context, two refinements reach the error of TTT-MLP's block approximation in less time, and 340M and 1.3B models match or exceed Gated DeltaNet and Mamba-2 on perplexity and recall.

The study asks whether a decoder-only language model needs an independently trainable input vector per token: with V=65,536 and d_model=1024, it replaces the usual V×d_model input table with fixed 16-bit binary token codes plus a parameter-free tiled lift to model width, removing 67.1M trainable parameters (about 12.5% of the untied learned-input baseline), and across three training seeds 32-layer models trained on roughly 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes, with the authors stating these descriptive results do not establish statistical superiority or equivalence.

The study asks whether a decoder-only language model needs an independently trainable input vector per token: with V=65,536 and d_model=1024, it replaces the usual V×d_model input table with fixed 16-bit binary token codes plus a parameter-free tiled lift to model width, removing 67.1M trainable parameters (about 12.5% of the untied learned-input baseline), and across three training seeds 32-layer models trained on roughly 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes, with the authors stating these descriptive results do not establish statistical superiority or equivalence.

The work proposes Hierarchical Continuous Diffusion Language Models (HC-DLM), in which a continuous latent is the only state persisting across steps while tokens are read out from it at every step and fed back as a scaffold conditioning the next latent update, with a training objective derived from a variational bound on the token likelihood; at matched model size it improves over discrete and continuous diffusion baselines in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B.

The work proposes Hierarchical Continuous Diffusion Language Models (HC-DLM), in which a continuous latent is the only state persisting across steps while tokens are read out from it at every step and fed back as a scaffold conditioning the next latent update, with a training objective derived from a variational bound on the token likelihood; at matched model size it improves over discrete and continuous diffusion baselines in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B.

Without retraining, this work retains only the tokens with the highest attention weights at each head, layer, and query while keeping their original weights unchanged, and estimates the effective attention set size needed to stay within a chosen loss tolerance by measuring the increase in negative log-likelihood (NLL); it finds that relatively small selected sets keep NLL close to the full-attention baseline, that attention-based selection substantially outperforms random selection, that the required set size grows with context while its fraction of context decreases, that in BABILong experiments with a fixed annotated supporting fact additional background pushes support tokens down the attention ranking and reduces their attention mass, and that renormalizing the retained weights can subs

Without retraining, this work retains only the tokens with the highest attention weights at each head, layer, and query while keeping their original weights unchanged, and estimates the effective attention set size needed to stay within a chosen loss tolerance by measuring the increase in negative log-likelihood (NLL); it finds that relatively small selected sets keep NLL close to the full-attention baseline, that attention-based selection substantially outperforms random selection, that the required set size grows with context while its fraction of context decreases, that in BABILong experiments with a fixed annotated supporting fact additional background pushes support tokens down the attention ranking and reduces their attention mass, and that renormalizing the retained weights can subs

The work introduces FRAC, a selective state space model architecture derived from fractional dynamics that approximates a heavy-tailed fractional kernel with a finite-state, log-spaced sum of exponential modes, replacing exponential forgetting with power-law long memory and improving long-context performance over SSM baselines such as Mamba2, GDN, and Mamba3 on synthetic long-tail and recall tasks, 1.3B-parameter language modeling, and DNA modeling, while staying competitive on short-context tasks.

The work introduces FRAC, a selective state space model architecture derived from fractional dynamics that approximates a heavy-tailed fractional kernel with a finite-state, log-spaced sum of exponential modes, replacing exponential forgetting with power-law long memory and improving long-context performance over SSM baselines such as Mamba2, GDN, and Mamba3 on synthetic long-tail and recall tasks, 1.3B-parameter language modeling, and DNA modeling, while staying competitive on short-context tasks.

The work introduces WhiteMatter, which uses a learned router to mix past-token representations from any depth into shared key-value (KV) channels so every layer can draw on cross-layer information, together with cyclic Gauss–Seidel iteration to speed up training and prefill; under matched training-token budgets, half-cache WhiteMatter improves perplexity and average downstream scores over matched standard Transformers at two scales up to about 1.3B parameters, while the full-cache version performs comparably to a deeper standard Transformer.

The work introduces WhiteMatter, which uses a learned router to mix past-token representations from any depth into shared key-value (KV) channels so every layer can draw on cross-layer information, together with cyclic Gauss–Seidel iteration to speed up training and prefill; under matched training-token budgets, half-cache WhiteMatter improves perplexity and average downstream scores over matched standard Transformers at two scales up to about 1.3B parameters, while the full-cache version performs comparably to a deeper standard Transformer.

The work proposes FactorEngram, which replaces monolithic per-pattern n-gram embeddings with sparsity-regularized coefficients over a dictionary shared across patterns and reuses that same dictionary for basis-level contextual gating, improving language modeling, downstream tasks, and long-context retrieval on 340M- and 1B-parameter Transformer backbones.

The work proposes FactorEngram, which replaces monolithic per-pattern n-gram embeddings with sparsity-regularized coefficients over a dictionary shared across patterns and reuses that same dictionary for basis-level contextual gating, improving language modeling, downstream tasks, and long-context retrieval on 340M- and 1B-parameter Transformer backbones.