A 27B model learns to propose language-model upgrades: 200 real training runs beat GPT-6-sol on held-out tasks
Synopsis
Deciding which change is worth trying when a language model improves a language model has depended on costly training experiments. LM2LM uses a frozen LLM as a research world model that predicts a proposal's gain from real experimental records in context, turning those predictions into reinforcement-learning rewards, and periodically executes the current policy's top-rated proposals to write measured outcomes back into the records. Across eight autoregressive and diffusion language-model development environments, Qwen3.8-27B trained this way outperforms GPT-6-sol on held-out tasks, even when GPT-6-sol is given 300 real experimental records in context, at about 33% lower data-generation cost.
Interpretation
A language model can now learn, through reinforcement learning, to propose effective model improvements without running a full training job for every proposal. Previously this ability came mainly from in-context learning, placing past experimental records in the prompt; supervised fine-tuning or rejection-sampling fine-tuning on a few hundred past proposals did not consistently improve proposal quality. Under real-execution evaluation on held-out research states, LM2LM reaches Avg@4 of 17.06 on autoregressive tasks and 7.10 on diffusion tasks, the highest among compared methods, and beats the base model on 93.3% and 100% of states.
The training signal comes from a frozen LLM that reads real experimental records in context and predicts how much validation-loss improvement a proposal would yield. Scoring reinforcement learning with real execution would be costly: at roughly 45 B200 GPU-hours per training run, one GRPO-style setup would need about 25,600 proposal evaluations and over 1,152,000 GPU-hours. The research world model is instantiated as GPT-6-sol, the proposer backbone as Qwen3.8-27B and the builder as GPT-6-astra, evaluated on 40 training tasks and 10 held-out tasks with 200 real runs by default.
To keep predicted rewards from drifting as the policy changes, training three times selects the current policy's highest-rated proposals for real execution and writes their measured gains back into the context. A static variant with fixed records ends training with predicted advantages negatively aligned with real ones, and its pairwise ranking accuracy falls from 0.72 to 0.37, below chance. At the same budget of 200 real runs, top-rated calibration reaches autoregressive Avg@4 of 17.06 versus 14.76 for adding random records, while random calibration lowers it to 10.84.
Perspective
Evaluation covers four autoregressive families (OLMo-3, Qwen3, NanoChat, Marin) and four diffusion families (E2D2, E2D2-SW, MDM, BD3LM), with 40 training tasks and 10 held-out tasks; held-out states are built only from real experimental histories and are excluded from proposer training, world-model grounding and calibration. Results concern proposal-level improvements rather than end-to-end research campaigns. The authors propose extending the framework to broader research environments such as language-model post-training and testing it in longer, end-to-end campaigns.
World-model predictions remain imperfect, and on-policy calibration still requires a limited number of additional model-training runs. Per-family results are uneven: Avg@4 trails GPT-6-sol with in-context records on Marin and E2D2-SW, and headroom is limited on NanoChat. The authors also note that predicted rewards and experimental gains do not guarantee the safety or broader usefulness of proposed model changes.