AI agents meet real engineering drawings: the best configuration reaches 48.3% on a hidden test set
Synopsis
Mercor and Google's AI & Infrastructure research team released Physical-Engineering's Last Exam (PLE), 85 long-horizon tasks testing whether AI agents can do mechanical, electrical, civil and structural engineering work. Each task runs in a containerized project folder of about 61 files, including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and planted outdated revisions, with no internet access and up to six hours. The best configuration, Claude Opus 5.5 in Claude Code, reaches 48.3% on the hidden test set at an average of 27 million tokens and 63 minutes per task; its designs met 91% of individual constraints on average, but only 41% cleared every hard check.
Interpretation
Agent evaluation moves from office paperwork to the engineering work that builds the physical world, with mechanical, electrical, civil and structural disciplines each carrying their own codes, conventions and failure modes. Earlier agentic benchmarks measured office tasks such as building a financial model, reviewing a contract, drafting a deck or working through an inbox, as the APEX family does; physical engineering was barely measured at all. PLE holds 85 long-horizon tasks, 30 mechanical, 27 electrical, 18 civil and 10 structural, written and validated end to end by licensed engineers from Mercor's expert network.
Task environments deliberately reproduce the mess an engineer inherits on a project, so much of the difficulty lies in working out what the actual problem is. An engineer joining a project is not handed a clean problem statement, but inherits earlier revisions, half-finished calculations, client emails with shifting requirements and drawings that disagree with the spec. Each task runs in a containerized project folder of about 61 files, with no internet access and up to six hours to find governing codes and physical limits, run calculations and deliver something a licensed reviewer would accept.
Grading puts cost efficiency behind hard safety and feasibility checks, so an agent earns cost-efficiency points only after its design clears every hard check. In software a 90% solution can ship as a beta, whereas in physical engineering a 90% solution fails review. Across 14 frontier models and seven agent harnesses, Claude Opus 5.5 leads at 48.3%, followed by GPT 6 Astra at 45.8% and Gemini 4 Argon at 40.6%; model choice matters far more than harness choice, with scores across models spanning up to 50 points while the same model on different harnesses moves a median of about 3.
Perspective
The evaluation covers 85 long-horizon tasks across mechanical, electrical, civil and structural engineering, with agents working in containerized project folders without internet access for up to six hours; Google has released 29 public tasks, the grading tools and the evaluation container, and the open competition runs on the hidden test set.
Results are framed as frontier models not yet being fully trusted with physical engineering work: the top configuration reaches 48.3% on the hidden test set, and its designs met 91% of individual constraints but only 41% cleared every hard check; competition and prize details are not yet announced.