PaperBenchX moves paper reproduction into 12 disciplines: the best agent configuration clears the 90% bar on only 14% of tasks
Synopsis
PaperBenchX extends paper reproduction beyond machine learning to 12 research areas, requiring agents to rebuild key experiments inside native scientific software such as HFSS, Lumerical, Meep and PySCF, then replaying each submitted workflow from scratch and scoring only regenerated evidence. Across 93 tasks, ten frontier agent configurations score a mean rubric value between 27.53% and 59.08%, and the strongest fully reproduces only 14% of tasks at the 90% threshold, with modeling and execution scores well above validation.
Interpretation
Cross-disciplinary paper reproduction becomes a scorable agent task: the agent receives the paper, a public task specification and a domain-native scientific environment, and must reconstruct a key experiment and deliver a runnable workflow. Earlier reproduction benchmarks largely focused on AI and machine learning, where experiments share relatively homogeneous software stacks, while cross-domain benchmarks typically isolate only parts of the workflow. The benchmark contains 93 expert-constructed tasks spanning 12 research areas and 10 domain-native scientific computing environments, with 3,168 scorable criteria.
Scoring looks not at the agent's report but at evidence regenerated by a from-scratch replay. A final output or code that merely runs cannot distinguish a scientifically correct implementation from a plausible one. Evaluation clears generated outputs, restores only the submitted package, runs reproduce.sh with private evaluation resources inaccessible and outbound network disabled, freezes replay artifacts as a read-only evidence set, and has the same GPT-5.6-sol evaluator judge each hidden rubric leaf.
Agents make clear progress in modeling and execution, but composing partial progress into full reproduction remains hard. Average modeling and execution scores reach 62.73% and 61.83%, while validation falls to 42.91%. Only 80 of 930 outcomes exceed the 90% threshold, concentrated on 22 tasks; 71 tasks are completed by no configuration, and of 304 outcomes scoring at least 80% on both modeling and execution only 53 cross the full-reproduction threshold.
Perspective
The evaluation covers 93 tasks across 12 research areas and 10 domain-native environments, with 10 configurations run on the same task set; one representative task per area is released publicly, nine with redistributable scientific computing environments and three with only task specifications and scoring details because of commercial licenses.
The results are described as current agents making substantial progress in modeling and execution while validation remains their weakest stage, and more interaction turns do not necessarily improve reproduction; these conclusions are limited to the evaluated task set and configurations.