Paper2Bench turns published papers into auditable exams: the best agents match or beat every baseline on 80% of tasks

Published alignment benchmark studies were rewritten as tasks an agent can execute directly, with agents seeing only sanitized expression matrices, spatial coordinates and the required output schema while ground-truth coordinates, cell-type annotations and scoring scripts stay outside the workspace. Earlier scientific agent benchmarks focused largely on code generation and bug fixing graded by automated tests, while scientific tasks are often loosely defined, multi-dimensional in success and expensive in ground truth; existing paper-based benchmarks typically grade against static reference values reported in the papers rather than re-running published baselines under a unified scorer. Tasks derive from the SABench and GEASO alignment studies and span 19 datasets and five task families, 121 tasks in total, of which 32 have ground-truth coordinates, 69 have cell-type or anatomical region annotations, and the rest are scored by gene expression similarity.

Re-executing baselines under one scorer, while rebuilding the original inputs, exposed a coordinate unit mismatch in a published evaluation: input and ground-truth coordinates of the Drosophila embryo stack differ by 20-fold. That mismatch gave even the stationary reference slice a fixed error floor, while methods that shrank coordinates toward the origin appeared artificially accurate. After rescaling ground truth into native input pixel units, the artifact is resolved and methods separate cleanly, with PASTE cutting error by about 75% relative to unaligned inputs.

Agents mostly solved alignments by writing compact numerical optimization code themselves rather than calling domain packages; pre-installing alignment packages and literature recommendations changed their tool use and raised cost without consistently improving accuracy. Under the basic prompt, 94% of agent runs solved alignments from first principles with custom code; under the rich-context prompt, 89% of runs executed a published package and 54% kept one in the final submission. Rich context raised token usage and cost by about 1.5-fold on average, while mean within-task percentile changes pointed in different directions across models, with claude-opus-5.5 and gemini-4-argon sitting at opposite ends of the reported range.

Distilling agent runs into a deterministic Python program that makes no LLM calls still beat most published baselines on held-out tasks. Before distillation, agent solutions depended on repeated proprietary model calls and were hard to reproduce or reuse; the distilled program runs repeatedly at zero inference cost. Under random splits the distilled program averaged above 87% of the 16 reference baselines, matching interactive frontier agents and exceeding the median published package at 54%; under task-family splits it reached 78% overall but dropped to 30% on unseen simulated tissues.

The evaluation is limited to spatial transcriptomics slice alignment, with tasks drawn from the SABench and GEASO studies, the whole-mouse-brain MERFISH atlas and Albis-simulated tissues; the authors release the Paper2Bench pipeline, all tasks, the containerized evaluation harness and a trace explorer, and note that extending the pipeline to more open-ended biological analyses requires domain-specific data contracts and validation checks.

Frontier models and command-line harnesses evolve quickly, so rankings reflect the vendors' default configurations at the time and may change as new models ship and older ones are retired; more exhaustive hyperparameter tuning of baseline packages or prompt engineering could raise individual scores, while the evaluation intentionally reflects the limited tuning budget scientists actually have.