Youran Sun

A new benchmark turns senior researchers' corrections of scientific AI agents into 248 two-way judgment questions: given only the project record from before the correction, four of five frontier models more often chose the original plan that a human had overruled, topping out at 63.5% against 73.4% for fresh researchers. The cases come from 26 real projects across six fields run over roughly half a year, and each asks a model to choose between the original plan and the human-corrected alternative.

A new benchmark turns senior researchers' corrections of scientific AI agents into 248 two-way judgment questions: given only the project record from before the correction, four of five frontier models more often chose the original plan that a human had overruled, topping out at 63.5% against 73.4% for fresh researchers. The cases come from 26 real projects across six fields run over roughly half a year, and each asks a model to choose between the original plan and the human-corrected alternative.