World models move to closed loop: a Transfer-Auto reference run on four GPUs returns seven 720p videos and 99 LiDAR sweeps
Synopsis
NVIDIA has packaged world-foundation-model training, optimization and inference into Cosmos Framework, with agent-callable skills and checkpoints that run out of the box. Developers previously had to fit a model to their own sensors, action space and data, then get it into an application, two separate rework efforts. The released Cosmos 3 Nano simulation recipes cover driving, robotics and surgery: a Transfer-Auto reference run on four GPUs returns seven 720p videos and 99 LiDAR sweeps, while Sim-Auto, distilled into a causal few-step model, generates observations step by step from driving-policy input and returns them to the policy for closed-loop evaluation.
Interpretation
NVIDIA has folded world-model training, optimization and inference into Cosmos Framework, an open and fully modular framework whose skills and recipes an AI agent can call directly. Developers working with world models face two separate problems: fitting a model to their own sensors, tasks and data, and getting that model into an application reliably, each previously built from scratch. Support now reaches past Cosmos 3 to NVIDIA Alpamayo, GR00T, Qwen, pi0.5 and other compatible Hugging Face models, and the released Cosmos 3 Nano recipes ship runnable example checkpoints for driving, robot and surgical simulation.
The simulation checkpoints take an editable description of the scene: a World Scenario Map writes lanes, objects, road markings, traffic signs and road boundaries as controls, and the prompt supplies weather, lighting, time of day and road conditions. Cosmos3-Nano-Transfer-Auto uses bidirectional generation to denoise a whole multi-view clip at once for offline synthetic data, while Cosmos3-Nano-Sim-Auto is its causal few-step-distilled variant, streaming frames autoregressively as actions arrive. The reference run pairs seven synchronized WSM videos with per-camera captions, first-frame RGB conditioning and HD-map LiDAR controls, and on four GPUs returns seven 720p videos of 297 frames at 30 FPS plus 99 LiDAR sweeps.
Perspective
The demonstrated scope is three simulation settings: multi-camera plus LiDAR generation and closed-loop rollouts for driving, two-arm robot interaction and depth-video-to-RGB conversion, and a surgical world model post-trained on Open-H-Embodiment. The checkpoints support PyTorch and vLLM-Omni and were tested on NVIDIA Hopper and Blackwell GPUs. The stated starting point is to pick a simulation checkpoint that matches the task, reproduce the reference workflow, and adapt it to your own sensors, actions and data.
The speedup figure is a measured mean of roughly 2–3x for speculative decoding on H200, limited to that GPU and that decoding configuration. Deployment carries its own prerequisite: the throughput profile needs 2, 4 or 8 compatible GPUs, and changing only NIM_PERF_PROFILE to throughput in the single-GPU example will fail profile selection.