Robot data without a teleoperator: an agent compiles one calibration clip into a simulation task, reaching 93 versus 84 real-robot successes

Collecting robot demonstration data no longer has to be done by a person driving the arm; a coding agent can take the teleoperator's role and turn one data request plus one calibration clip into multi-view video data a policy trains on directly. Teleoperation had been the main source of robot policy training data, with every episode costing robot and operator time and a reset after failures, and every new task, scene or viewpoint starting from scratch. On five real bimanual tasks, policies trained only on farm data reached 93 successes in 100 trials in the original scene against 84 for policies trained on as many teleoperated episodes, and under a tablecloth and coloured light absent from both training sets they kept 87 against 46.

The simulator supplies actions and states while a video model conditioned on real images supplies appearance, and separating the two yields frames that follow the simulated motion and look like real camera recordings. Earlier real-to-sim-to-real pipelines depended on reconstructed scene assets, which represent cloth and lit displays poorly, while letting a video model synthesize the motion itself gives up exact action labels. Across the five tasks, H3 renders deviate from the simulated optical flow by 4.4 pixels on average and agree on 65% of moving pixels, whereas pairing the same renders with the simulator video half a second later raises the error to 8.8 pixels and lowers agreement to 41%; head-view frames sit at a KID of 0.83 to real camera frames against 1.93 for the simulator's own renderings.

Data production is split into stages that check one another: the expert controller is developed against a judge it cannot inspect, and every rendered episode passes automatic checks and an independent reviewer before admission. Letting the model that writes a task also write its success check amounts to approving its own mistakes, and a video model may copy the layout of its references or hallucinate objects. Before use, each verifier specification is calibrated on deliberately corrupted episodes, with objects deleted, duplicated or enlarged, events shifted in time, views swapped or colours changed, and every corruption must be caught by the responsible check; a separate reviewer agent then compares each episode with the real reference frames and the simulated state.

The demonstrated scope is five bimanual tasks covering rigid, articulated, deformable and display-bearing objects, evaluated on a dual-arm PiPER robot with 20 real-robot trials per task and condition; the cost figures come from the authors' self-hosted H3 rendering deployment and a comparison against a teleoperation quote from China.

The method depends on a simulator that can reproduce each task's skills, and a task the available simulator cannot handle needs a new simulator variant first; dynamics a calibration clip does not exhibit, such as mass or friction, are labelled as assumed and randomized over, which may not suffice for tasks whose success depends on precise dynamics; rendered episodes can still contain defects that automatic checks miss, which is why every episode undergoes independent review and rejected renders count toward cost; the evaluation covers one dual-arm embodiment and five tasks.