ML agents skip the training run: a 4B world model predicts program outcomes on 77.9% of held-out programs
Synopsis
AI research agents must train and evaluate a model for every candidate program, so execution cost caps how many candidates they can compare per step. MLE-World, finetuned from Qwen3.5-4B, reads a program's code and data description and predicts its terminal output, exception type and validation score, plus a confidence score. On held-out programs from 17 MLE-bench Lite tasks its predicted terminal output matches the real run on 77.9% of programs versus 41.1% for the strongest prompted baseline, and search guided by it reaches up to 22.1% higher observed mean performance than the agent without a world model while using less real execution.
Interpretation
Machine-learning engineering agents can now compare several candidate programs before running only one of them for real. Earlier pruning and ranking methods cut total execution but leave the cost per execution unchanged and return no error traces or training logs to guide later debugging. On 1,700 held-out programs from 17 MLE-bench Lite tasks, MLE-World's predicted terminal output matches the real run on 77.9% of programs, against 41.1% for DeepSeek, the strongest prompted baseline.
The prediction costs one inference call rather than a training run, so an agent can compare more candidates within a fixed time budget. MLE-World is a full-parameter supervised finetune of Qwen3.5-4B on roughly 497,000 logged pairs of code and execution output. The input carries the program, installed packages and versions, a data overview and the time limit; the output decodes an exception class as a single added token first, then terminal output and validation score.
A confidence score shipped with each prediction decides which candidates need a real run. Predictions are not always right: on held-out programs 22% of predicted terminal outputs and 14% of predicted exception classes disagree with the real run. The gate uses separate thresholds for predicted successes and failures and never runs programs predicted to time out; across the five policies it cuts real execution time by 30% to 40% while broadly maintaining performance.
Perspective
The main evaluation is in distribution: held-out programs are new, but all 17 tasks appear in the finetuning data. Prediction accuracy, the five-policy search comparison and the gating savings are measured in that setting; execution prediction on three tasks excluded from finetuning is reported separately.
Prediction errors can still misdirect search, timeouts are the least accurately predicted outcome, and a missed timeout still costs a full run. On the UCB policy the gate lowers mean percentile by 0.9, which the authors describe as broadly maintaining performance. Transfer evidence on tasks excluded from finetuning mainly concerns recognizing execution failures; predicting successful training outcomes and timeouts remains less established.