Robot pretraining drops paired robot data: cross-video prediction from human demos lifts unseen-layout success from 18% to 41%

World-action pretraining no longer predicts the next frame only inside one demonstration; it uses one human demonstration to predict the actions and future frames of another demonstration of the same task. Earlier video pretraining predicted future frames within a single demonstration, where context and target share scene appearance, viewpoint and collection source, letting prediction exploit demonstration-specific visual cues instead of separating transferable interactions. On the MimicDroid L1 to L3 split, cross-video pretraining raises Cosmos Policy's average success from 40.9% to 68.3% and Cosmos3 from 59.8% to 77.7%, with gains at all three levels and the largest on the unseen layouts.

The model receives the current query frame and state together with a reference demonstration's video and actions, then jointly predicts the query's own actions and future frames. The reference supplies an example of task progression while the query observation specifies the target scene and source, so predicting the query future requires transferring the demonstrated interaction into the query's visual and geometric context. Training pairs come from same-task demonstrations across different collection sources in EgoVerse, giving 10.99M query-reference pairs of 2-second windows, with 10.52 references per query on average and 6.35 of them cross-source.

Cross-video pretraining aligns action-slot features for the same motion across people, recording rigs and robots, and concentrates attention on the acting hand and the manipulated object. The original video backbone spreads attention over edges and background on both human demonstrations and robot observations rather than concentrating on the interacting hand and object. On held-out EgoVerse and RoboMIND windows, cross-video pretraining reaches 0.039 action MSE and 68.5% accuracy against 0.046 and 55.6% for within-video pretraining; the held-out-embodiment regression reaches Spearman correlations of 0.71 on human and 0.62 on robot windows, against 0.55 and 0.41 for the original video model.

The demonstrated scope covers two video backbones, three benchmarks and both in-context and context-free post-training protocols, with actions written in one Cartesian task space anchored at each episode's start. The method targets robot policies using end-effector controllers; the paper states all three post-training targets are Cartesian and that joint-space policies have not been shown. Pairing needs task labels and hand tracking, so unlabeled video or video without hand pose needs a label-free gate and a pose estimator. The paper lists extending pairing beyond one human dataset to less structured sources as future work.

The paper reports pretraining on paired windows from a single human dataset, and the scaling curve has not saturated: a quarter of the corpus already matches within-video pretraining on the full corpus on the unseen levels. Sharing one Cartesian task space between human and robot actions is a precondition, and the paper notes a joint-space action target would break pairing, the action prior and post-training reuse at once. In the representation analyses, the cross-video model additionally receives a matched reference's future actions, an input that bears directly on a future-motion probe.