RoboTryOn swaps humans for robots in demo videos: new-task success rises from 2.1% to 60.0% with no teleoperated data

First-person human video can now become robot demonstrations with action labels, and the policy runs on the raw camera view at test time without masking or overlaying the live image. Both prior routes retarget reconstructed human hand motion into robot actions and narrow the appearance gap by masking the arm or overlaying a rendered robot, applying the same edit to the camera stream during training and deployment. Across eleven tabletop tasks on two robots, RoboTryOn raises in-domain success from 47.3% to 75.9% on the OpenArm and from 69.6% to 79.5% on the RB-Y1; on seven out-of-domain tasks with no robot demonstrations, where both baselines stay below 10%, it raises success from 3.1% to 73.0% on the OpenArm and from 1.0% to 46.9% on the RB-Y1.

The pipeline has three stages: translate selected keyframes into robot keyframes, synthesize the robot video between them, then infer joint actions from that video. Keyframe translation follows video virtual try-on, conditioned on an arm-and-hand mask and a hand skeleton and autoregressively referencing the previously translated keyframe; the video model is guided between keyframes by hand skeleton flow. The inverse dynamics model uses a Depth Anything V2 backbone with a flow-matching action head, takes two frames one second apart and predicts a 20-step joint action chunk; all three generative models are trained only on teleoperated robot data, while inference extracts the same mask and skeleton conditions from human video.

Action labels read from synthesized video are more executable than retargeted ones, which is what lets the policy learn out-of-domain tasks. Retargeting maps the reconstructed human hand onto the robot hand, but hand reconstruction is imprecise and human and robot hands differ in morphology and kinematics, so grip force cannot come from shape mapping. Replaying action labels open loop in simulation completes 69.7% of episodes with RoboTryOn labels versus 34.8% with retargeted labels, with the gap concentrated at grasping; at pretrained weights, the flow-matching gradient cosine similarity between RoboTryOn data and teleoperated data is 0.53, against 0.17 and 0.14 for the baselines.

The demonstrated scope is eleven tabletop tasks on two real humanoid robots with dexterous hands (an OpenArm with Inspire RH56F1 hands and an RB-Y1 with WujiHand2 hands), with human videos recorded from the robot head-camera position and the neck and base held fixed in teleoperated data; the authors also translate human videos into a GR-1 but could not evaluate policies on it without the physical robot. The authors propose scaling to large egocentric corpora such as EgoDex for every embodiment they own, and improving inverse dynamics model accuracy.

The action labels are pseudo labels: the inverse dynamics model reads joints from pixels and hand reconstruction supplies the wrist position, so they cannot capture forces and their precision is bounded by stereo video and hand reconstruction, leaving force-dependent or tight-insertion tasks not yet labeled accurately; the wrist correction also relies on head-camera calibration. With a moving camera the input falls outside the training distribution of the video model and the inverse dynamics model, which always predicts fixed neck and base actions because those never move in its training data. The demonstrated motion must be one the robot can execute; the pipeline can retime but cannot correct a reach beyond the arm or a grasp the robot hand cannot form. Synthesis trades compute for collection time, favoring tasks expensive to teleoperate, and no method completes the Air Fryer task on the RB-Y1.