Robot arms miss grasps by millimeters; Auriga recovers 68.3% of the planning-execution gap
Synopsis
Vision-language models can read a task yet still fumble the grasp: on RoboDojo, removing arm execution raises the same model's success rate from 22.5% to 76.2%, locating the bottleneck in physical execution. Auriga separates model decisions from robot execution, with the model keeping an explicit task belief and every motion passing a single admission gateway that previews the trajectory, checks stage tolerances during execution, and stops on divergence. Frozen for evaluation, Auriga reaches 57.7% success and 64.0 Score across 42 tasks, ranking first, and 73.2% on RoboCasa365.
Interpretation
Auriga attributes vision-language model failures in manipulation mainly to physical execution rather than task judgment, and hands measurement, checking and stopping to a task-agnostic execution harness. On 210 RoboDojo diagnostic episodes, the same model succeeds in 76.2% when allowed to move objects along its own waypoints without the arm, but in only 22.5% when driving the arm through the benchmark's reference end-effector interface. On that diagnostic set, covering all 42 tasks with five episodes each, Auriga closes 68.3% of the gap, reaching 59.2% success against 22.5% for the reference interface on the same episodes.
The harness makes no task decisions; the model maintains a revisable task belief recording its current plan, open questions, and the status of a few task-critical premises. Step-wise and chunked action interfaces issue one motion from the latest frames, so the model never sees the route before committing, while code-as-policy hands judgment to a script that cannot see the scene and may exit normally after failed motions. Auriga has the model describe a complete multi-stage dual-arm sequence in one call, which the harness compiles into a trajectory preview showing path, per-stage cost and clearance warnings while simulation is paused, for the model to review before execution.
Every physical motion passes one admission gateway, which checks model-set arrival tolerances and gripper openings during execution and stops the robot when a stage leaves its tolerance. Under the reference interface the arm misses grasps by a few millimeters, loses objects in flight, or hits container rims, and step-wise interfaces notice only on the next frame, after the arm has moved. The stop receipt reports completed stages, measured deviations, unexecuted stages and a fresh camera view, so the model recovers in place from the current physical state instead of resetting the scene.
The frozen harness ranks first overall on the RoboDojo leaderboard with 64.0 Score and 57.7% success, and stays first under scene randomization. The same model with the reference harness scores 29.0 with 22.5% success; on the 12 generalization tasks Auriga reaches 52.28% success on standard scenes and 45.61% on random scenes, while learned policies drop by up to 33.9 percentage points under randomization. Evaluation covers 42 tasks over three seeds, 6,300 episodes, with the harness frozen and no target-task exploration, demonstrations or resets; on RoboCasa365's 50 Target50 tasks over 250 episodes the same frozen harness reaches 73.2% overall success.
Perspective
The demonstrated scope is zero-shot single-episode evaluation on two simulated benchmarks: 42 dual-arm tabletop tasks on RoboDojo and 50 Target50 mobile manipulation tasks on RoboCasa365, with the harness frozen during evaluation and no target-task exploration, demonstrations or resets.
The authors attribute failures to the boundary of kinematic trajectory execution: stage verification and belief updates reliably detect discrepancies, but recovery actions that disturb coupled multi-body structures can exhaust the episode budget, as in stack_bowls failing at the 800-step limit and HeatKebabSandwich at the 100-request limit; proposed next directions are continuous force-sensitive manipulation, lower deliberation latency and automated cross-task tool synthesis.