# Scaling Law

e-means replaces the k-means centroid update with a mini-batch exponential moving average and uses fitted scaling laws to set batch size and decay directly from the problem configuration, so no hyperparameter sweep is needed. Across embedding datasets it reaches the loss Lloyd k-means attains at 25 epochs in 1.4 to 2.6 epochs, translating to 9.5 to 45.5 times wall-clock speedups over Flash-KMeans while converging to a lower loss. On DINO10B, the exact-assignment version partitions 10 billion vectors into 16 million clusters in 2.5 epochs at a loss below Lloyd at 25 epochs.

e-means replaces the k-means centroid update with a mini-batch exponential moving average and uses fitted scaling laws to set batch size and decay directly from the problem configuration, so no hyperparameter sweep is needed. Across embedding datasets it reaches the loss Lloyd k-means attains at 25 epochs in 1.4 to 2.6 epochs, translating to 9.5 to 45.5 times wall-clock speedups over Flash-KMeans while converging to a lower loss. On DINO10B, the exact-assignment version partitions 10 billion vectors into 16 million clusters in 2.5 epochs at a loss below Lloyd at 25 epochs.

The Platonic Representation Hypothesis, which holds that separately trained models converge on a shared picture of the world, becomes a theorem for JEPAs (joint-embedding predictive architectures): if each modality's positive pairs link the same world states, the encoders agree up to one rotation. The resulting good-view versus bad-view distinction predicts alignment, and a 109M-parameter text JEPA reaches .184 on the PRH benchmark, above every language model evaluated, up to Qwen2.5-72B at .154.

The Platonic Representation Hypothesis, which holds that separately trained models converge on a shared picture of the world, becomes a theorem for JEPAs (joint-embedding predictive architectures): if each modality's positive pairs link the same world states, the encoders agree up to one rotation. The resulting good-view versus bad-view distinction predicts alignment, and a 109M-parameter text JEPA reaches .184 on the PRH benchmark, above every language model evaluated, up to Qwen2.5-72B at .154.

Visual generation is split into two stages: autoregressive pretraining acquires visual knowledge, and diffusion post-training renders it into pixels. In controlled comparisons at roughly 400M parameters, the autoregressive-then-diffusion route reached 65.5 on GenEval, above 61.2 for pixel diffusion trained from scratch, and a semantic tokenizer needed neither an image decoder nor a reconstruction loss to support rendering. Allocating compute by fitted scaling laws, a roughly 3B-parameter text-to-image model trained in about 6,600 H100 GPU-hours scored 0.86 on GenEval and 87.78 on DPG-Bench.

Visual generation is split into two stages: autoregressive pretraining acquires visual knowledge, and diffusion post-training renders it into pixels. In controlled comparisons at roughly 400M parameters, the autoregressive-then-diffusion route reached 65.5 on GenEval, above 61.2 for pixel diffusion trained from scratch, and a semantic tokenizer needed neither an image decoder nor a reconstruction loss to support rendering. Allocating compute by fitted scaling laws, a roughly 3B-parameter text-to-image model trained in about 6,600 H100 GPU-hours scored 0.86 on GenEval and 87.78 on DPG-Bench.

Using EditLens and Pangram to label Common Crawl web data from 2021 to 2026, the authors find that 27.5% of tokens passing FineWeb quality filtering in June 2026 are AI-generated, rising to 31.1% in August, and by pretraining 800 models from 19.9M to 973M parameters while varying the ratio of added AI to human tokens they fit a new scaling law with separate benefit and harm terms: AI tokens lower loss for data-starved models before saturating and reversing into harm, raise loss almost immediately for Chinchilla-optimal models, and the law predicts held-out sizes with lower error than eleven existing laws, implying that training at August 2026's AI share takes 1.6x the compute.

Using EditLens and Pangram to label Common Crawl web data from 2021 to 2026, the authors find that 27.5% of tokens passing FineWeb quality filtering in June 2026 are AI-generated, rising to 31.1% in August, and by pretraining 800 models from 19.9M to 973M parameters while varying the ratio of added AI to human tokens they fit a new scaling law with separate benefit and harm terms: AI tokens lower loss for data-starved models before saturating and reversing into harm, raise loss almost immediately for Chinchilla-optimal models, and the law predicts held-out sizes with lower error than eleven existing laws, implying that training at August 2026's AI share takes 1.6x the compute.

The work introduces Loop Scaling Laws, the first scaling law that jointly models recurrence and MoE sparsity alongside model size and training data, using a bounded, sparsity-conditional recurrence mapping to characterize the effective-parameter gain from looping and its saturation; it predicts held-out loss more accurately than linear and power-law mappings, guides the choice of recurrence and expert count under compute and memory budgets, and at trillion-token scale lets a 0.3B-active/1.3B-total looped MoE match a roughly 2x larger non-looped MoE on reasoning benchmarks.

The work introduces Loop Scaling Laws, the first scaling law that jointly models recurrence and MoE sparsity alongside model size and training data, using a bounded, sparsity-conditional recurrence mapping to characterize the effective-parameter gain from looping and its saturation; it predicts held-out loss more accurately than linear and power-law mappings, guides the choice of recurrence and expert count under compute and memory budgets, and at trillion-token scale lets a 0.3B-active/1.3B-total looped MoE match a roughly 2x larger non-looped MoE on reasoning benchmarks.

The study systematically characterizes the scaling properties of same-family on-policy distillation (OPD) on Qwen2.5 (0.5B–14B) math reasoning, finding a regular early useful-transfer regime in which held-out accuracy rises approximately linearly with the square root of token-level reverse KL from the student initialization, and fitting power laws in student scale, teacher scale, and teacher gold score that predict peak accuracy and transfer rate, with every weak-to-strong student peaking above its own teacher.

The study systematically characterizes the scaling properties of same-family on-policy distillation (OPD) on Qwen2.5 (0.5B–14B) math reasoning, finding a regular early useful-transfer regime in which held-out accuracy rises approximately linearly with the square root of token-level reverse KL from the student initialization, and fitting power laws in student scale, teacher scale, and teacher gold score that predict peak accuracy and transfer rate, with every weak-to-strong student peaking above its own teacher.