NVIDIA releases Cosmos3-H-Dreams, a real-time generative surgical simulator trained on clinical cholecystectomy recordings

Generative surgical simulation now covers clinical cholecystectomy inside the body rather than tabletop suturing, with instruments moving in the generated scene in response to control commands. The earlier Cosmos-H-Dreams delivered interactive generative simulation on dVRK tabletop suturing, a foundational proving ground where instruments work against a relatively simple background. The new model learns from video and robot kinematics recorded during clinical procedures on the CMR Surgical Versius platform; the release describes it as the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.

Conditioned on an initial image and tool actions, the model generates successive blocks of video while keeping tissue deformation, occlusion and instrument appearance consistent over long rollouts. Rather than explicitly modeling scene geometry, tissue deformation and physical interactions, the approach learns visual dynamics directly from synchronized video and robot kinematics. Training uses 832 x 480 video with actions aligned at 10 Hz across 28 action dimensions covering both arms' pose and gripper, energy controls, and thumbstick axes and buttons; the teacher is first adapted on 24 future frames and then extended to 72 frames, 2.4 and 7.2 seconds at 10 Hz.

Real-time interaction depends on distilling a bidirectional teacher into a causal student that generates video in short blocks as new actions arrive, with a streaming runtime managing attention history over long rollouts. Errors accumulate during long rollouts and gradually alter instrument geometry or anatomical structure, so the student must be trained on its own generated context. Causal teacher forcing prepares sequential generation from recorded context, and Self-Forcing then trains the student on its own generated context guided by the teacher and a learned critic; FlashDreams supplies the accelerated streaming runtime, and an adapted FadeMem module consolidates the growing key/value cache during long rollouts.

The demonstrated scope is real-time generated rollouts of clinical cholecystectomy driven by two controlled instruments on a single NVIDIA RTX PRO 6000 GPU; the release states it is a research simulator, not a validated system for patient care.

Visual realism is a starting point for validation, not proof of biomechanical accuracy, and the generated scene must also follow translation, rotation and gripper commands. Fewer denoising steps speed inference at the expense of lower visual quality, and the two checkpoints let developers trade off between them.