Robot world models stop learning only from success: training on human egocentric video and perturbed failures aligns predicted success with real deployment

Action-conditioned world models can now learn from egocentric human manipulation video and failure actions, not only from a robot's own successful demonstrations. Robot datasets are dominated by successful demonstrations with limited coverage of failures and off-target interactions, while egocentric human video is hard to use because human hands and robot arms differ in form. On 100 held-out Franka trajectories from DROID and 100 real-world failed trajectories, HF-WM reaches PSNR 26.077 and FVD 14.449 against OSCAR's 23.357 and 31.815, with closely comparable performance on successful and failed executions.

The model maps human and robot actions into one shared skeleton representation, then uses stabilized point trajectories to make the generated video attend to genuine interaction motion. Human hands and robot arms differ in degrees of freedom, dimensionality, and motion structure, and egocentric recordings carry camera jitter that obscures interaction-relevant regions. Training uses roughly 45K Franka trajectories and 83K human interaction trajectories, and after camera-motion compensation the coherent drift of background points is substantially reduced while the relative motion of the hand and manipulated objects is preserved.

A second stage perturbs successful actions into counterfactual ones and applies GRPO post-training with action alignment and physical plausibility rewards, learning failure dynamics without paired failure videos. Failure videos have no ground truth to supervise, so direct supervised fine-tuning cannot cover the consequences of abnormal gripper operations, mistimed actuation, and end-effector pose deviations. The physical plausibility reward alone tends to produce nearly static scenes or motion that contradicts the conditioning actions, the action alignment reward alone compromises temporal consistency in background regions, and combining the two rewards gives the best metrics.

The method is fine-tuned in two stages on roughly 45K Franka trajectories and 83K human interaction trajectories, evaluated on 100 held-out DROID Franka trajectories and 100 real-world failed trajectories, and used as a policy evaluator over six generalist policies on the same 90 matched execution sessions. It is aimed at researchers and engineers working on robot world models, policy evaluation, and cross-embodiment data use.

Failure dynamics are learned from perturbations of successful actions rather than real failure videos, so the model learns plausible consequences under counterfactual conditions. The physical plausibility reward is scored by a vision-language model against a tiered checklist, and any Tier-1 failure caps that candidate's reward. The strong correlation between predicted and real-world success rates comes from six policies over 90 matched sessions, and the ranking error arises from a predicted tie between two policies whose real-world success counts differ by one.