# Language model pretraining

4 related articles

Subscribe to updates

arXiv Oct 26, 2026
LLM pretraining optimizers stop orthogonalizing layer by layer: TUON stacks layers into a tensor and beats Muon and TEON on four model scales
Large language model pretraining optimizers are moving from handling each weight matrix separately to stacking the momenta of one module's layers into a third-order tensor. Layer-wise Muon ignores cross-layer correlation, while matricization-based tensor optimizers handle only two-layer stacks and QKV. TUON applies a unitary transform along the layer mode, polarizes each transformed slice, and inverts the transform; it lowers validation loss versus Muon and TEON on GPT-124M/350M and Qwen-0.6B/4B at comparable step time.
Large language model pretraining optimizers are moving from handling each weight matrix separately to stacking the momenta of one module's layers into a third-order tensor. Layer-wise Muon ignores cross-layer correlation, while matricization-based tensor optimizers handle only two-layer stacks and QKV. TUON applies a unitary transform along the layer mode, polarizes each transformed slice, and inverts the transform; it lowers validation loss versus Muon and TEON on GPT-124M/350M and Qwen-0.6B/4B at comparable step time.
Peking University
Anda Tang（First author）
TUON Tensor-Aware Optimizer DecorQ Transform Tensor-SVD Language model pretraining
arXiv Oct 12, 2026
Low-resource languages stop losing the expert pool: a category bias on routing, removable at inference
COAX adds a positive routing bias toward a designated expert subset for each data category during pretraining, letting languages, modalities, or domains with tiny data shares claim a larger expert share; across MoE models from 345M to 10.6B, low-resource categories lose less, while inference stays standard top-k routing.
Ayush K Tarun（First author）
COAX Mixture of Experts Category-aligned expert specialization Language model pretraining
arXiv Oct 1, 2026
Across 800 pretrained models, wild AI web text raises loss once data is plentiful, and a 31.1% AI share costs 1.6x the compute
Using EditLens and Pangram to label Common Crawl web data from 2021 to 2026, the authors find that 27.5% of tokens passing FineWeb quality filtering in June 2026 are AI-generated, rising to 31.1% in August, and by pretraining 800 models from 19.9M to 973M parameters while varying the ratio of added AI to human tokens they fit a new scaling law with separate benefit and harm terms: AI tokens lower loss for data-starved models before saturating and reversing into harm, raise loss almost immediately for Chinchilla-optimal models, and the law predicts held-out sizes with lower error than eleven existing laws, implying that training at August 2026's AI share takes 1.6x the compute.
Using EditLens and Pangram to label Common Crawl web data from 2021 to 2026, the authors find that 27.5% of tokens passing FineWeb quality filtering in June 2026 are AI-generated, rising to 31.1% in August, and by pretraining 800 models from 19.9M to 973M parameters while varying the ratio of added AI to human tokens they fit a new scaling law with separate benefit and harm terms: AI tokens lower loss for data-starved models before saturating and reversing into harm, raise loss almost immediately for Chinchilla-optimal models, and the law predicts held-out sizes with lower error than eleven existing laws, implying that training at August 2026's AI share takes 1.6x the compute.
University of Maryland, College Park
Jenna Russell（First author）
Scaling Law AI-generated web text WildAI Language model pretraining
arXiv Oct 1, 2026
Scaling synthetic pre-pretraining from 1B to 7B and 100B tokens keeps the token savings, but the authors trace them to long-range retrieval rather than a grammatical prior
The study systematically evaluates synthetic pre-pretraining (PPT) across four parameter scales from 500M to 7B, four pre-training data mixtures, five PPT tasks, and pre-training budgets up to 100B tokens, finding that downstream performance and token-efficiency gains persist at scale (saving at least 21B pre-training tokens at 3B) but that there is no consistent evidence the gains come from a grammatical prior; instead they arise from PPT tasks that improve long-range retrieval, are insensitive to code and math share, and diminish only when web text is absent.
The study systematically evaluates synthetic pre-pretraining (PPT) across four parameter scales from 500M to 7B, four pre-training data mixtures, five PPT tasks, and pre-training budgets up to 100B tokens, finding that downstream performance and token-efficiency gains persist at scale (saving at least 21B pre-training tokens at 3B) but that there is no consistent evidence the gains come from a grammatical prior; instead they arise from PPT tasks that improve long-range retrieval, are insensitive to code and math share, and diminish only when web text is absent.
Mohamed bin Zayed University of Artificial Intelligence Hugging Face University of Sheffield
Atsuki Yamaguchi（First author）
Synthetic Pre-pretraining Long-range Retrieval Language model pretraining

You’re all caught up

LLM pretraining optimizers stop orthogonalizing layer by layer: TUON stacks layers into a tensor and beats Muon and TEON on four model scales

Low-resource languages stop losing the expert pool: a category bias on routing, removable at inference

Across 800 pretrained models, wild AI web text raises loss once data is plentiful, and a 31.1% AI share costs 1.6x the compute

Scaling synthetic pre-pretraining from 1B to 7B and 100B tokens keeps the token savings, but the authors trace them to long-range retrieval rather than a grammatical prior