A humanoid catches thrown boxes with onboard vision alone: 65% at 2.5 m, 76% of human catchers

A humanoid can now catch a human-thrown box using only its onboard camera, without motion capture or marked objects. Earlier humanoid catching systems mostly relied on motion capture or retroreflective objects, or used only end-effectors, and rarely combined whole-body balance, fast prediction, and onboard vision. In simulation, egocentric RGB-D input catches 85.7% of throws, compared with 87.9% for the same decoder given privileged box observations, so visual prediction preserves most of the catching performance.

The capability is trained in three steps, each learning one thing, linked by a compact representation of object dynamics. Learning control end-to-end from pixels is expensive and hard to optimize, because every rollout needs RGB-D rendering while the policy explores both a large visual space and a 29-dimensional action space. Stage 1 learns whole-body catching from box state with reinforcement learning, Stage 2 learns a box-dynamics latent from delayed and incomplete observations, and Stage 3 regresses that same latent from egocentric RGB-D history while the controller stays fixed.

On hardware, catch success falls with distance, but at close range it approaches human catchers. Human catchers succeed on 85.0%, 68.3%, and 28.3% of throws at 2.5, 5, and 7 m. The real Unitree G1 catches 65% and 40% at 2.5 and 5 m, which is 76% and 59% of human success at those distances, with 20 trials per condition.

The system catches 85.7% of throws from egocentric RGB-D in simulation, and 65% and 40% on the real robot at 2.5 and 5 m, with 20 trials per condition. The authors treat catching as representative of a broader class of visually reactive whole-body skills, and explicitly name extending this decomposition to other such tasks as a natural next step.

Long-range visual grounding is the main limitation: on 5 m throws, faster than any training throw and beyond the 3.0–3.5 m Stage 3 data, visual success falls to 43.0% while privileged input reaches 79.7%; adding far throws to the Stage 3 data raises simulated 5 m success to 75.8%, not yet tested on hardware. The authors also report that the controller is effectively size-blind, that hardware latency was measured only on their detector-based pipeline and exceeds the randomized range, and that rendering omits motion blur and exposure dynamics.