Humanoid HALO learns perceptive whole-body motion in simulation, then hits 91% over 360 trials in six unmapped real homes
Synopsis
Humanoids asked to sit down or walk to a named object need paired supervision mapping pixels and free-form language to whole-body joint commands, and that data has so far come from teleoperation or wearable capture, where an hour of data costs at least an hour of human time. HALO moves data generation into simulation, synthesizing egocentric vision-language-motion episodes across a thousand procedurally generated homes, then uses a layered stack that splits a free-form request into subtasks, generates five-second whole-body motion plans, and executes them with a perceptive tracker. Without seeing a real home in training, it transfers zero-shot to six unscanned, unmapped real homes, succeeding in 91% of 360 trials.
Interpretation
Whole-body humanoid data no longer has to be demonstrated episode by episode; the training corpus can be generated in simulation at the scale of available compute. Teleoperation and wearable capture previously tied data production to physical labor, so an hour of data cost at least an hour of human time. The training corpus spans a thousand procedurally generated homes, none of which anyone had to visit or scan.
The system turns one free-form language request into continuous motion on the robot, with perception used both to decide what to do and to decide how to move. A high-level orchestrator calls an off-the-shelf vision-language model every two seconds to decompose the request into a sequence of subtasks, each naming a skill and its object. A mid-level generator maps the subtask and an egocentric RGB image at 3 Hz to a five-second plan of joint angles and pelvis poses, and a 50 Hz tracker combines that plan with onboard depth and proprioception to issue joint commands.
In real homes never seen during training and never scanned or mapped, the system executes six whole-body skills directly. Prior work reported limited trials in physical environments not seen during training, and some systems were evaluated only inside pre-scanned rooms. Across 360 physical trials in six real home environments, mean single-task success was 91% in simulation and 91% in the real world, including 85% for sitting, 93% for side-stepping through a half-open door, and 100% for crouching under an obstacle.
Perspective
The demonstrated scope is six single-task skills and three multi-task chains on a Unitree G1 across 360 physical trials in six unscanned, unmapped real homes; adding a new physical task still requires a person to write its prompt and author the keyframes of its motion sequence.
Chained multi-task success falls below the product of single-task rates, and failures often occur because the next subtask begins with its target unobserved; perception is largely visual, so once an object leaves the camera's field of view or is occluded the policy has no signal that it is still there, and contact-rich tasks degrade.