SWAM carries video priors into robot actions: 61.56% success on randomized scenes, 12.86 points above OpenWAM
Synopsis
Shanghai Jiao Tong University and collaborators assembled a roughly 47,000-hour multisource pool and extended a pretrained video model's language–video priors into language–video–action learning, yielding the world action model SWAM. On the 50 manipulation tasks of RoboTwin 2.0, SWAM reaches 61.56% success under the randomized Hard setting, 12.86 percentage points above the strongest baseline OpenWAM, and 88.10% under the clean Easy setting, slightly below OpenWAM's 89.30%. On a real bimanual platform, its three-task average Hard success rate is 40.00% against OpenWAM's 23.33%.
Interpretation
A world action model predicts video changes and robot actions together during training, so the language–video correspondences learned in video pretraining remain usable for action generation instead of being displaced when action supervision arrives. Earlier world action model work concentrated on architectures and training mechanisms, and video-pretraining methods generally treated language–video alignment as an established foundation; how to preserve that prior and address language–video–action misalignment after adding action learning had not been systematically examined. On the 50 RoboTwin 2.0 tasks, SWAM reaches 61.56% success under the randomized Hard setting, 12.86 percentage points above the strongest baseline OpenWAM at 48.70%; its Easy setting is 88.10% against OpenWAM's 89.30%, so the two are close under clean conditions.
The model pairs a video branch initialized from Wan2.2-TI2V-5B with a randomly initialized action branch, each keeping its own residual stream, with information passing only from video to action. The action branch has no direct text cross-attention; language reaches it only through attention interactions with the video branch, reusing the language-to-visual-dynamics correspondences built during video pretraining. Under the same 30k-step multisource pretraining, the dual-stream model with this language conditioning pathway reaches 57.36% Hard success, 4.88 percentage points above the standard dual-stream model at 52.48%; the single-stream model reaches 52.56%.
Multisource experience divides labor by supervision type: general and egocentric videos supply language–video supervision, while UMI and robot demonstrations additionally supply state and action supervision, and samples without valid action labels contribute only to video learning. Under a fixed 20k-step budget, sampling egocentric videos and UMI data at 1:1 gives 53.84% Hard success, above UMI-only at 51.28% and egocentric-only at 42.56%. Raising UMI action-label coverage from 0% to 100% lifts Hard success from 46.72% to 51.28%, a gain of 4.56 percentage points, with success rising at all five coverage levels.
Perspective
Evaluation covers the 50 manipulation tasks of RoboTwin 2.0 and three tasks on a bimanual Flexiv Rizon 4 platform: cup stacking, towel folding and gift-in-box; simulation runs 100 episodes per task and setting, and the real world uses ten fixed layouts per task and setting with three trials each, 30 trials in total. The authors state that data, code and model weights will be released.
The authors note limited support for complex reasoning and explicit multi-stage planning, and say the limited historical context may hinder long-range tasks; they plan memory mechanisms and hierarchical planning as extensions. Pretraining gains continue only within the evaluated range, with real-world Hard success rate and partial success rate rising as training video hours grow from 22.5k to 90k.