Runway turns a video model into a robot controller: Praxis-1 solves 46.2% of RoboLab-120 and ships open weights
Synopsis
Runway released Praxis-1, which converts a 27B-parameter video foundation model into a world action model that predicts both the next scene and the motion to reach it, and ships it as open weights. Robot data is scarce and expensive to collect, and generalist policies have leaned on teleoperation or egocentric footage; Praxis-1 instead pretrains on everyday video, then mid-trains on embodied video and jointly with an action expert. Fine-tuned on DROID, it solved 555 of 1,200 RoboLab-120 episodes (46.2%), the highest among world action models and open-source robot foundation models on that leaderboard.
Interpretation
The pretraining source for robot policies moves from robot-specific data to everyday video: a video foundation model is turned into a world action model that returns both a predicted future scene and actions on each call. Real robot data is scarce and gathering enough to train a generalist policy is expensive, and some labs have responded with more teleoperation or more egocentric footage. Praxis-1 fine-tuned on DROID solved 555 of 1,200 episodes (46.2%) under RoboLab-120's official protocol of 10 trials per task, above FLUX 3 Action at 42.9% and above every open-weight VLA on the board, including π0.5 and GR00T N1.6.
The model works in three parts: a video backbone predicts the next stretch of scene, an action expert reads the backbone's internal state to work out the motion that produces it, and thin per-robot projections translate poses and actions into a shared format. The action expert is a dense 16-layer transformer of width 1024 with about 314M parameters, and its cross-attention layers each read one of eight backbone encoder blocks, so it reads the future as the backbone imagines it. Each call takes an instruction, the latest camera frames and the current pose, and returns a predicted video chunk plus 1.6 seconds of actions, 24 rows at 15 Hz; with default chunked execution a call takes about 0.55–0.6 seconds round trip on one H100.
Perspective
The demonstrated scope is simulation and real-station tasks on the DROID-fine-tuned single Franka arm, the bimanual YAM, RT-1 and SO-101, plus RoboLab-120's official evaluation protocol; the guidance settings were tuned for the DROID Franka, and other robots need their own.
More mid-training compute led to better fine-tunes on both the seen and unseen robots, but not at every step, and the authors say the exact relationship needs further study; in offline evaluation the deployment error mixes future-video and future-action prediction, which the authors note is not a fully reliable metric.