Scientific agents stall on judgment: four of five frontier models pick the plan a senior researcher overruled
Synopsis
A new benchmark turns senior researchers' corrections of scientific AI agents into 248 two-way judgment questions: given only the project record from before the correction, four of five frontier models more often chose the original plan that a human had overruled, topping out at 63.5% against 73.4% for fresh researchers. The cases come from 26 real projects across six fields run over roughly half a year, and each asks a model to choose between the original plan and the human-corrected alternative.
Interpretation
Scientific agents can keep experiments running for weeks or months, yet senior researchers still had to step in and judge whether to scale up or first check a measurement, and whether a failed attempt undermined the idea or only its implementation. Automated checks and review still depended on models noticing mistaken assumptions themselves, and accumulating rules became harder for models to follow reliably, so these judgments stayed with people. The interventions come from 26 projects across six scientific fields run over roughly half a year with publication as the goal, spanning theory, simulation, computation and analysis of existing data.
The benchmark reconstructs each human correction as a decision point before it happened, asking a model to choose between the original plan and the human-corrected alternative. An evaluator sees only the project record at that commit, including code, experimental logs and progress notes, and not the later history. Grok 4.6 wrote the 248 cases from full project histories, and cases were retained using 120 model responses per case so that model responses vary; the writer showed no advantage on its own cases in a controlled pilot.
At medium reasoning effort with the project record, four models chose the overruled plan more often than the corrected one, and the record made them worse than seeing the two plans alone. Mean accuracy across the five models is 39.2% with the project record and 55.8% without it, and the paired interval excludes zero for four models. The highest observed project score is Astra's 63.5% at xhigh, below the 73.4% human baseline; on 49 cases all five models chose the overruled plan in both option orders.
Perspective
The evaluation covers theoretical and computational research using CPUs and GPUs, including simulation and analysis of existing data such as biological images; wet-lab execution and workflows using specialized commercial software such as COMSOL remain outside it. The cases test judgments central to PhD research, including experiment selection, anomaly diagnosis and revision of conclusions. The benchmark measures recognition of corrective decisions, while generating plans and carrying out research require separate evaluation.
The selected cases concentrate on decisions where model responses vary, so scores are diagnostic and do not estimate how often these failures occur across all scientific activity. Several cases come from each project, so project-cluster intervals are reported alongside case-level ones. Evaluated clients expose different reading tools, and requested effort settings do not specify equal token budgets. The second review repairs 264 errors while introducing 305, and mean accuracy falls from 44.0% to 42.3%.