NVIDIA opens four Cosmos 3 checkpoints so driving and dual-arm robots can act and see the next frame

Driving and robotics researchers can now download four open-weight Cosmos 3 checkpoints and use them to generate observations that change with actions, rather than relying only on fixed recorded data. The Cosmos world-model development platform previously offered the Cosmos 3 model family; this release adds four open-weight checkpoints together with post-training recipes. All four models are post-trained examples of Cosmos 3 Nano with 16B trainable parameters each, released under OpenMDW1.1 alongside checkpoint downloads, a technical report and demos.

Streaming generation connects video prediction to ongoing interaction: the model predicts each short segment from the prompt, available controls and the history it has already generated. Cosmos 3's Mixture-of-Transformers backbone links separate text-processing and visual-generation networks through attention, and temporal-causal attention reads history and controls while blocking future target observations. The model works on compressed video representations called latents, where each latent time step after the initial image represents four RGB frames; Transfer predicts one step per chunk and the optimized Bimanual setup in the report predicts two.

The four checkpoints divide the work: Transfer-Auto produces offline synthetic driving data, Sim-Auto is its causally distilled streaming variant, Sim-Transfer renders simulator controls into RGB, and Sim-Bimanual predicts the next observation directly from dual-arm actions. Sim-Auto uses generated history and updated driving controls to produce successive chunks of synchronized camera video and LiDAR in just a few denoising steps. Transfer-Auto generates synchronized video across up to 11 camera views and supports joint RGB-LiDAR generation; Sim-Bimanual supports the YAM and AgiBot dual-arm embodiments, driven by a policy or human-operated joysticks.

The demonstrated scope is inference and post-training for four open-weight checkpoints: Transfer-Auto for offline synthetic driving data, Sim-Auto for streaming driving simulation, Sim-Transfer for rendering RGB from simulator controls, and Sim-Bimanual for the YAM and AgiBot dual-arm embodiments. Running them requires Linux, an NVIDIA GPU, Docker and the NVIDIA Container Toolkit; the published vLLM-Omni setup was tested on one GB200, which NVIDIA states is not a minimum-memory specification.

NVIDIA states that measuring where these models faithfully support policy learning and evaluation, and understanding their limits, remains to be done, and that improving where they fall short is the work ahead; a successful short training run verifies execution, not convergence, and quality must be evaluated on held-out episodes.