Surgical simulation learns from real gallbladder surgery video: one GPU, real-time response to instrument motion

Generative surgical simulation now covers in-body clinical cholecystectomy imagery, where instrument actions drive how the generated video continues. The previous Cosmos-H-Dreams, built on Cosmos-Predict2.5, offered interactive generative simulation only for dVRK tabletop suturing, a foundational proving ground. The new model learns from video and robot kinematics recorded during clinical procedures; to the release team's knowledge it is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.

The model does not explicitly model scene geometry, tissue deformation or physical interaction; it learns visual dynamics directly from synchronized video and robot kinematics. Numerical simulation writes out geometry, deformation and contact explicitly, while the generative route replaces that with scene evolution learned from data. Training samples require two visible engaged instruments, a left Cadiere grasper and a right monopolar hook, and limit camera commands to emphasize instrument-driven interaction; the action space has 28 dimensions covering both arms' pose and gripper, energy controls, thumbstick axes and buttons, with 832 x 480 video and actions aligned at 10 Hz.

For generation inside a live control loop, the bidirectional teacher is distilled into a causal student that emits video in blocks as new actions arrive, paired with a streaming inference runtime. The bidirectional teacher is first adapted on 24 future frames, then extended to 72 frames, 2.4 and 7.2 seconds at 10 Hz, giving longer sequences for learning surgical dynamics. Causal teacher forcing prepares the student for sequential generation from recorded context, and Self-Forcing trains it on its own generated context guided by the teacher and a learned critic to address error accumulation; the first student uses four denoising steps per block, and the two-step checkpoint is separately trained rather than the four-step model run with fewer steps.

The demonstrated scope is cholecystectomy, two controlled instruments, and real-time generation on a single RTX PRO 6000 GPU, with four-step and two-step checkpoints trading visual quality against inference cost. The release team lists as next directions covering all phases of cholecystectomy, combining learned realism with explicit geometry, contact and tissue mechanics, evaluation with surgeons, support for more instruments, and closed-loop training of autonomous surgical policies.

The release team notes that visual realism is a starting point for validation, not proof of biomechanical accuracy, and that scenes must follow translation, rotation and gripper commands. Small errors can accumulate over long rollouts and gradually change instrument geometry or anatomy, and the four-step versus two-step tradeoff buys faster inference at lower visual quality; all listed directions are research plans, not established clinical capabilities.