Coding agents take their first physical exam: the best setup solves only 34% of 456 core tasks under real-robot-like conditions
Synopsis
The Embodied First Exam connects general coding agents to 47 simulated robots and lets each agent choose direct control, code to control, or training a policy, while the real-robot-like Standard Setting supplies only sensor observations and, by default, no resets. Among eight agents, GPT-6 Astra in Codex solves 34% of the Standard tests on 456 core tasks, while on the same 391 tasks it reaches 77% in the Privileged Setting, which exposes the full simulator state and allows unlimited resets. The exam also labels the stage at which failed attempts go wrong, so failures can be compared across settings and agents.
Interpretation
General coding agents now have a unified robotics exam on which they decide for themselves how to control the robot, instead of being fixed to the input-output interface of visuomotor policy learning. Earlier robotics benchmarks were built around visuomotor policy learning, where the agent chooses from an action space the benchmark defines, while digital coding-agent evaluations impose no physical-world constraints. The exam contains 1,156 tasks in 35 suites on 47 robots, including 9 new suites with 411 tasks that cover spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control and sim-to-sim controller adaptation.
Each task can run in up to three settings that differ only in what the agent observes, whether it can reset, and whether the world keeps running, never in which control approach it may choose. The Standard Setting mimics a real robot: the agent observes only through the robot's sensors and by default cannot reset, whereas the Privileged Setting gives the full simulator state and unlimited resets to search for an approximate upper bound. On the 391 core tasks that have both settings, GPT-6 Astra in Codex solves 77% in the Privileged Setting and only 30% in the Standard Setting; on the 456 core tasks it solves 34% of the Standard tests.
Failures can be located at a specific stage of a task, so failures become comparable across agents and settings. The verifier decides whether an attempt passed but not where it went wrong, so the exam splits 427 core tasks into stage graphs and labels the failure stage of each failed attempt. In the Standard Setting, the concentration of failure stages over the eight agents is 61%, against 37% for a random baseline, and the most common failure stage is the task's first stage in 65% of the 264 Standard tests that have a single most common stage.
In controlled runs, adding back the two capabilities the Privileged Setting provides separates which one drives the gain. The Standard Setting lacks the exact state of every object and unlimited resets, and the two had not been measured apart. GPT-6 Astra's success rate on the 391 tasks rises from 28% to 45% with an object-state query and to 53% with unlimited resets and saved states, and to 68% with both, against 77% on its Privileged tests; at precise stages, resets solve 36% of 91 failures and the state query alone 21%.
Perspective
The exam covers 35 suites, 47 robots and 1,156 tasks, with results reported on 901 tests of 456 core tasks, all in simulation; the real-robot portion is a buzz wire game with one robot arm and is not counted in any success rate.
Each agent makes one attempt per test, so the results cannot separate agent ability from variation between runs; the core set excludes tasks judged easy and tasks solved by the screening agent GPT-6 Luna, which belongs to the same model family as GPT-6 Astra and may bias the core set against that family; the stage labels were written by agents, and the main writer, GPT-6.1 Sol, is also from the GPT-6 Astra family.