# agentic benchmarks

Spatial transcriptomics alignment lacked an auditable way to evaluate AI agents, so Paper2Bench turned two published alignment benchmark studies into 121 executable tasks, had agents solve them in containers with ground truth withheld, and re-ran 16 published baselines under one scorer. Across the 121 tasks, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks; distilling agent runs into a standalone program beat 87% of baselines on held-out tasks.

Spatial transcriptomics alignment lacked an auditable way to evaluate AI agents, so Paper2Bench turned two published alignment benchmark studies into 121 executable tasks, had agents solve them in containers with ground truth withheld, and re-ran 16 published baselines under one scorer. Across the 121 tasks, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks; distilling agent runs into a standalone program beat 87% of baselines on held-out tasks.

Mercor and Google's AI & Infrastructure research team released Physical-Engineering's Last Exam (PLE), 85 long-horizon tasks testing whether AI agents can do mechanical, electrical, civil and structural engineering work. Each task runs in a containerized project folder of about 61 files, including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and planted outdated revisions, with no internet access and up to six hours. The best configuration, Claude Opus 5.5 in Claude Code, reaches 48.3% on the hidden test set at an average of 27 million tokens and 63 minutes per task; its designs met 91% of individual constraints on average, but only 41% cleared every hard check.

Mercor and Google's AI & Infrastructure research team released Physical-Engineering's Last Exam (PLE), 85 long-horizon tasks testing whether AI agents can do mechanical, electrical, civil and structural engineering work. Each task runs in a containerized project folder of about 61 files, including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and planted outdated revisions, with no internet access and up to six hours. The best configuration, Claude Opus 5.5 in Claude Code, reaches 48.3% on the hidden test set at an average of 27 million tokens and 63 minutes per task; its designs met 91% of individual constraints on average, but only 41% cleared every hard check.

Agentic large language models handle complex tasks, yet long-horizon strategic reasoning lacks a dissectable benchmark. SpireBench has 20 LLMs play the complete commercial deck-builder Slay the Spire at its highest difficulty: frontier models beat heuristic baselines and match the average human player but remain far behind experts, and the gap widens as runs lengthen, with the strongest models winning act one as often as experts before falling behind. Memories models write across runs do not improve them and are often wrong and overly cautious.

Agentic large language models handle complex tasks, yet long-horizon strategic reasoning lacks a dissectable benchmark. SpireBench has 20 LLMs play the complete commercial deck-builder Slay the Spire at its highest difficulty: frontier models beat heuristic baselines and match the average human player but remain far behind experts, and the gap widens as runs lengthen, with the strongest models winning act one as often as experts before falling behind. Memories models write across runs do not improve them and are often wrong and overly cautious.

The work finds that pre-trained LLMs with a light inference harness, despite far lower single-shot accuracy (pass@1), often surpass their post-trained counterparts in solution coverage (pass@K) given sufficient test-time budget; it proposes the Sharpening Tax metric to quantify post-training's loss in test-time scalability, shows across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases) that the tax is prevalent, estimable from a few rollouts, and correlates with other metrics, and presents PTGS, a difficulty-adaptive Bayesian sampler that reduces the tax.

The work finds that pre-trained LLMs with a light inference harness, despite far lower single-shot accuracy (pass@1), often surpass their post-trained counterparts in solution coverage (pass@K) given sufficient test-time budget; it proposes the Sharpening Tax metric to quantify post-training's loss in test-time scalability, shows across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases) that the tax is prevalent, estimable from a few rollouts, and correlates with other metrics, and presents PTGS, a difficulty-adaptive Bayesian sampler that reduces the tax.

The work formalizes AI agent performance as an 'intent-execution' gap (the mismatch between what a model intends and what the harness executes), introduces a customizable harness called Simple Strands Agent (SSA) to align different model families, reproduces or improves the pass@1 reported by diverse model-provider families on SWE-Pro, SWE-Verified, Terminal-Bench-2, and Deep-SWE, and, based on 154k trajectories generated by SSA, represents agent trajectories in code state-spaces to compute solution distance and backtracking, using finer-grained metrics such as edit frequency, testing activity, and phase-transitions to reveal differences in problem-solving behavior and effort allocation across frontier models whose pass@1 numbers are relatively even.

The work formalizes AI agent performance as an 'intent-execution' gap (the mismatch between what a model intends and what the harness executes), introduces a customizable harness called Simple Strands Agent (SSA) to align different model families, reproduces or improves the pass@1 reported by diverse model-provider families on SWE-Pro, SWE-Verified, Terminal-Bench-2, and Deep-SWE, and, based on 154k trajectories generated by SSA, represents agent trajectories in code state-spaces to compute solution distance and backtracking, using finer-grained metrics such as edit frequency, testing activity, and phase-transitions to reveal differences in problem-solving behavior and effort allocation across frontier models whose pass@1 numbers are relatively even.