Robot world models stop aligning joints: shared latent actions turn human video into training data
Synopsis
RynnWorld-Latent conditions a video world model on continuous latent actions extracted by a frozen RynnLAM encoder, giving human, robot and simulation video one shared interface without aligning native control spaces. Pretrained on 23 datasets totaling about 553.5M frames and then post-trained per target robot, it reports better video-prediction metrics on public benchmarks and two real platforms, and data from latent-action synthesis and digital teleoperation raises downstream policy success in the evaluated settings.
Interpretation
A robot video world model can now be trained on an action description that is independent of the hardware, so egocentric human video, robot demonstrations and simulation rollouts enter one pretraining objective. World models conditioned on raw robot actions were tied to each platform's joint counts, coordinate conventions and control frequencies, and human video carries no native robot control labels. The pretraining corpus spans 23 datasets and roughly 553.5M frames, with human interaction video, robot demonstrations and simulation rollouts relabeled into one latent-action space.
The model predicts future video from one initial observation and a sequence of continuous latent actions, with no text conditioning and no common native action space. Latent actions come from a frozen RynnLAM encoder that reads visual change between adjacent frames, describing what changed rather than what command was issued. Each latent action is index-aligned with one future video latent because the latent-action stride equals the video VAE's temporal compression factor, which is what supports frame-aligned injection.
Action conditioning enters the pretrained video diffusion transformer through two paths: global action tokens in joint self-attention, and a frame-aligned local path that adds each action descriptor to its corresponding future video latent. The backbone has no adaptive layer normalization, so conditioning must arrive as attention tokens or explicit feature modulation; both added paths are zero-initialized and act as identity maps at the start. On Marvin-WUJI, removing frame-aligned injection drops PSNR by 0.72 and removing local modulation by 0.24; on Astribot-S1 the drops are 0.88 and 0.25, and the full model scores best on both platforms.
After post-training the model serves as a data engine: digital teleoperation retargets operator motion into target robot commands and generates paired action-video episodes, while latent-action synthesis edits a source video's first frame and reuses its latent-action sequence to generate a rollout under a new embodiment. Collecting real robot data in new environments is costly, and asset-based simulation needs explicit 3D assets and manual system identification. Synthetic mid-training raises LIBERO success from 94.4 to 96.1 and LIBERO-Plus from 53.6 to 79.4; on real tasks, adding digital-teleoperation data improves success on all four tasks, with each reported rate based on 35 trials.
Perspective
Evaluation covers three DreamDojo public evaluation sets plus held-out demonstrations on two real platforms, the Marvin-WUJI dual-arm dexterous-hand setup and the Astribot-S1 wheeled dual-arm gripper setup; each real-task success rate is based on 35 trials. Results concern models after target-robot post-training and do not establish zero-shot transfer to an unadapted robot.
Latent actions are inferred from visual transitions and may encode camera motion or appearance as well as manipulation, so image metrics and qualitative action following do not establish physical correctness. In the synthesis route, retargeted actions and generated observations come from separate processes, and temporal alignment plus filtering cannot guarantee physical consistency. Synthetic mid-training adds both data and optimization compute, and the current comparisons do not separate the two.