Physics agents meet whole research projects: 56 real tasks, and the best setup passes only 46 of 168 runs
Synopsis
FrontierPhysics replaces fixed physics questions and paper reproduction with given solutions by 56 end-to-end tasks that 42 physicists built from their own research, spanning nine areas of theoretical, computational and experimental physics. An agent receives only a high-level prompt like a principal investigator's brief, then must review the literature, set a plan, run the research, and deliver both deterministically checkable numerical outputs and a manuscript. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts, and manual review of trajectories traces the main weaknesses to insufficiently rigorous reasoning and limited expert intuition rather than gaps in physics knowledge.
Interpretation
The object of evaluation shifts from closed questions to a full research sequence: literature review, research plan, execution, structured numerical outputs and a manuscript. Earlier physics benchmarks mostly tested fixed questions, isolated subtasks, or reproduction with the original paper supplied, so the research question and solution structure were largely given in advance. The 56 tasks were adapted by 42 physicists from projects they had personally run for at least two weeks, drawn from 26 institutions and nine physics domains; 18 underlying studies are published, four of them in Nature.
A rollout passes only if it clears both numerical verification and an expert-authored rubric. Closed-ended tasks with deterministic answers saturate quickly and lose signal, while judging only the manuscript cannot confirm whether the results are right. Deterministic scripts check computed quantities and simulation outputs; the rubric splits into binary blockers and non-blockers scored 0, 1 or 2 with weights from 1 to 10, and any failed blocker or verifier means no pass.
Frontier models still largely fail at complete physics research, and the failures are not about physics knowledge. On all 56 tasks, overall pass rates stay below 30% for every agent tested. Claude Opus 5.5 in Claude Code CLI has the highest pass rate at 27.4%, passing 46 of 168 rollouts; among 168 manually labeled GPT-6 Astra rollouts, insufficiently rigorous reasoning accounts for 48%, missing physics intuition 34% and literature synthesis 27%.
Perspective
The evaluation covers all 56 tasks of FrontierPhysics v0.1, with agents running in sandboxes under a uniform 10-hour cap and GPU access available for resource-heavy tasks; the manual failure-mode labeling covers only the 168 single-agent GPT-6 Astra rollouts.
The authors note that verifier and rubric defects may remain, and manual review found nine of the 168 rollouts were incorrectly failed, which underestimates the pass rate; other models or model-harness pairs may fail in ways outside the taxonomy, so the false-negative correction applies only to GPT-6 Astra. With few tasks per domain, differences between domains are uncertain.