Agents take on real physical-engineering projects: all 14 frontier models fall below 50% on PLE's private test set

Physical engineering now has an end-to-end benchmark and open competition for agents, with 85 long-horizon tasks across mechanical, electrical, civil and structural engineering. Frontier language models already score above 90% on many academic benchmarks, but those mostly test textbook-style questions, single tool calls or small self-contained design problems, and do not show whether an agent can produce safe, code-compliant, cost-effective designs under a licensed engineer's review. Mercor experts and licensed practicing engineers built the tasks: 44 were expert-authored, with engineers creating the project folders, prompts and grading rubrics, and 41 were co-authored, with an agent pipeline assembling the environment and grader from licensable engineering standards before engineers validated and increased the difficulty of each.

Instead of answering a single prompt, an agent works inside a containerized project folder that mirrors what a practicing engineer receives on the job. Each task holds an average of 61 files, 5,170 in total across the benchmark, including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and document revisions. The agent must reason through these files, find the governing building codes and physical limits, run calculations and generate production-ready deliverables a licensed reviewer would approve; the environment is built on Mercor's Archipelago.

Grading combines code compliance with cost efficiency, and a design earns points only after passing every hard safety check. On co-authored tasks the top-scoring agent configuration satisfied 91% of individual constraints, yet only 41.3% of its deliverables passed every hard safety and feasibility check. Fourteen frontier models ran on seven publicly available agent harnesses (Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands and Pi), and the highest-performing model on its native harness reached a mean score of 48.3% on the private test set.

The demonstrated scope is 85 long-horizon tasks across mechanical, electrical, civil and structural engineering, of which 29 tasks, grading scripts and the evaluation container are public while the rest form the private test set; baselines cover 14 frontier models on seven public agent harnesses.

The 48.3% figure is a mean for the top model on its native harness, and the private test set's task composition is not detailed in the material; the 91% individual-constraint satisfaction versus 41.3% all-check pass rate on co-authored tasks shows that meeting constraints one by one does not equal overall compliance. The material also notes harness choice significantly affects run cost but gives no cost figures.