Recursive self-improvement reaches frontier pre-training: 801-30B-A3B trains from scratch in three months and ranks second among base models
Synopsis
Recursive self-improvement had only been closed on post-training, where each trial costs GPU-hours and returns a verifiable score; a frontier pre-training run costs hundreds of thousands of GPU-hours, cannot be repeated, and has no single-score objective. A newly formed team at Shanghai Innovation Institute and Xinxu AI split pre-training into smaller closable loops, using agents for debugging, tiny-SFT and related measures to estimate post-training headroom, and quantified workflows for data tasks, producing the 30B-A3B base model 801 in three months. On 13 base benchmarks it averages 73.13, behind only the larger Qwen3.5-35B-A3B at 76.67.
Interpretation
Recursive self-improvement is moved into frontier pre-training: agents now act on the whole production line that starts from random initialization, not only on post-training. Public recursive self-improvement work had targeted post-training, starting from an existing base model, costing GPU-hours per trial and returning a verifiable score; a frontier pre-training run consumes hundreds of thousands of GPU-hours over months and is usually run only once. Two pilots stopped at 1T and 3T tokens and the main run used about 11.8T tokens; at a matched 1T-token budget the mean over 22 benchmarks rose from 47.78 in the initial pilot to 51.96 in the revised pilot and 56.83 in the main run.
The approach decomposes the pre-training loop, which cannot close end to end, into three kinds of smaller loops that can close, so each step leaves rerunnable evidence. Debugging loops relocate diagnosis onto the infrastructure and data stacks that can be rerun cheaply and measure whether an anomaly disappears; the headroom loop estimates post-training headroom with tiny-SFT, a learning-aptitude curve and a max-over-protocols estimate; experience-to-metric loops give classifier development, site discovery and crawl coverage explicit scores. These loops supported data and recipe revisions across the two pilots and main pre-training, and five cross-stack diagnoses spanning infrastructure, data and recipe were completed in about one month.
Agents also cross from consuming training data to producing it, turning real software pull requests into executable, verifiable agentic trajectories. Agentic data had largely served post-training at limited scale; here it is scaled to pre-training volume while preserving task diversity and quality. More than 84% of collected raw pull requests became executable tasks, 220K executable PRs were rolled out across nine agent harnesses, and layered filtering yielded 1.1M trajectories totaling more than 100B tokens, with 134B agentic tokens in total used in the long-context pre-training stage.
Perspective
The demonstrated scope is one 30B-A3B mixture-of-experts model pre-trained from scratch on about 11.8T tokens in three months, plus a lightweight supervised fine-tuning probe on top; the model is released as a base model and full post-training is left to a subsequent release. The authors state they will open-source 801-30B-A3B-Base and the lightweight SFT data for evaluating base-model post-training headroom.
The authors note that data and recipe changed together, so the score gains at 1T tokens reflect iterations supported by all three loop types rather than any single fix; the lightweight SFT was not repeated with multiple random seeds, so small differences between adjacent checkpoints need caution; in the agentic comparison, 6.33B tokens of lightweight SFT is set against complete post-training pipelines and reported only as a headroom reference, not a like-for-like comparison; and each optimizer-allocation setting is represented by a single training run.