PaperBenchX moves paper reproduction into 12 disciplines: the best agent configuration clears the 90% bar on only 14% of tasks
PaperBenchX extends paper reproduction beyond machine learning to 12 research areas, requiring agents to rebuild key experiments inside native scientific software such as HFSS, Lumerical, Meep and PySCF, then replaying each submitted workflow from scratch and scoring only regenerated evidence. Across 93 tasks, ten frontier agent configurations score a mean rubric value between 27.53% and 59.08%, and the strongest fully reproduces only 14% of tasks at the 90% threshold, with modeling and execution scores well above validation.