Siyuan Hu

Humanoids asked to sit down or walk to a named object need paired supervision mapping pixels and free-form language to whole-body joint commands, and that data has so far come from teleoperation or wearable capture, where an hour of data costs at least an hour of human time. HALO moves data generation into simulation, synthesizing egocentric vision-language-motion episodes across a thousand procedurally generated homes, then uses a layered stack that splits a free-form request into subtasks, generates five-second whole-body motion plans, and executes them with a perceptive tracker. Without seeing a real home in training, it transfers zero-shot to six unscanned, unmapped real homes, succeeding in 91% of 360 trials.

Humanoids asked to sit down or walk to a named object need paired supervision mapping pixels and free-form language to whole-body joint commands, and that data has so far come from teleoperation or wearable capture, where an hour of data costs at least an hour of human time. HALO moves data generation into simulation, synthesizing egocentric vision-language-motion episodes across a thousand procedurally generated homes, then uses a layered stack that splits a free-form request into subtasks, generates five-second whole-body motion plans, and executes them with a perceptive tracker. Without seeing a real home in training, it transfers zero-shot to six unscanned, unmapped real homes, succeeding in 91% of 360 trials.