WorldBridge splits video prediction from action: one labeled demo per new task lifts success from 55.90% to 70.70%
Synopsis
WorldBridge separates video prediction and action generation into two experts and uses structured attention to route predicted future frames into actions, then tests what video and action supervision each contribute on RoboTwin 2.0. With only in-distribution video-action supervision, out-of-distribution task success is 25.07%; adding videos of those tasks without action labels raises it to 55.90%, and one action-labeled demonstration per task raises it to 70.70%. Under the full data setting, WorldBridge reaches 96.23% average success on RoboTwin 2.0.
Interpretation
WorldBridge separates executing a new robot task into two examinable parts: predicting how the new task's visuals will unfold, and translating that prediction into executable actions. Prior embodied generalization work focused more on changes in execution conditions such as object arrangement, camera viewpoint, lighting and appearance, whereas a new task requires behavior beyond the training tasks. On RoboTwin 2.0, training with only in-distribution video-action supervision gives 25.07% success on out-of-distribution tasks and 91.85% in-distribution.
The model pairs a video expert with an action expert; each keeps its own parameters, they exchange information through joint self-attention, and they are trained with joint flow matching. A structured attention mask in the post-training stage builds a one-way path from memory and current observations through future video features to action predictions, making the predicted visuals an explicit condition for control. Under the full RoboTwin 2.0 setting, WorldBridge reaches 96.16% on Clean and 96.30% on Randomized scenes, averaging 96.23%; under Clean2Random, trained on Clean data only, it averages 80.9%.
A unified data pipeline maps heterogeneous robot trajectories and egocentric human-hand demonstrations into a shared 46-dimensional action representation, with dimension-wise validity masks preserving partial supervision. Robot platforms differ in control interfaces and annotation availability, so naive mixing would lose semantics. The curated corpus contains 1,275,715 robot episodes (9,729.58 hours) and 2,105,846 egocentric episodes (12,591.63 hours), totaling 3,381,561 episodes and 22,321.21 hours.
Perspective
The demonstrated scope is simulated manipulation evaluation on RoboTwin 2.0, RoboCasa GR1, RMBench, DOMINO Level-1, SimplerEnv-WidowX and RoboDojo, plus six real-robot tasks across the two platforms AgiBot G2 and ARX AC-1, with 20 trials per task. On real robots WorldBridge averages 72.5% success, against 57.5%, 39.2%, 29.2% and 10.8% for the compared methods.
The authors state that extending the video-to-action path to more diverse tasks with less action supervision remains future work. On real robots, sorting screws is WorldBridge's lowest-success task at 55%, which the authors list as a remaining precise small-part sorting challenge in that task suite. On DOMINO, the Randomized configuration reaches 56.1% success versus 59.1% on Clean, which the authors describe as a modest decrease under additional scene variation.