Nvidia Releases an AI Surgical Simulator Trained on Real Surgical Recordings

Nvidia unveiled Cosmos3-H-Dreams, a surgical simulator, on October 20 local time. The AI was trained on real cholecystectomy (gallbladder removal) footage along with recordings of a surgical robot’s movements. When a surgeon moves an instrument, the model generates video of the surgical scene that would follow. Nvidia says real-time interaction runs on a single NVIDIA RTX PRO 6000 GPU.

From practice drills to real surgery

The previous model, Cosmos-H-Dreams, handled only benchtop scenes: suturing drills performed with the dVRK surgical robot on a training rig. The new model learned from recordings of actual surgeries carried out with CMR Surgical’s Versius surgical robot. Moving beyond a training setup with a simple background, its scope now extends to the motion of tissue inside the body and to regions occluded by instruments.

How it works differs from conventional simulation, which computes object geometry and tissue deformation piece by piece. Instead, the model learns the relationship between video and robot motion in advance, then takes only a starting image and instrument control inputs to generate the video that follows. Inside the body, even small movements stretch tissue and change how light reflects. Nvidia explained that the model has to render such changes while continuously preserving anatomical structure and instrument shape.

Move the instrument, and the scene changes

For training data, the team selected scenes in which a left-hand grasper and a right-hand hook are both visible and actively involved in the task. Camera motion was restricted so the model would concentrate on learning the effects of instrument movement. The control data, which captures the position and orientation of both robot arms along with grasper state, was aligned to the video at 10 times per second.

Image: huggingface.co

Nvidia also released demonstrations applying different instrument controls to the same starting frame. Driven by inputs recorded through a VR headset, the surgical scene unfolds differently in each case. The company says such demonstrations let you check three things: whether the instrument moves as commanded, whether tissue responds in step with that movement, and whether tissue and instrument shapes hold up over time.

Fast version versus high-quality version: what’s the difference?

The released models come in a four-step and a two-step version, referring to the number of processing steps used to generate video. The two-step version produces video faster at the cost of image quality. It is not simply a pared-back configuration but a separately fine-tuned model. Nvidia also applied a training approach that reduces the problem of errors accumulating when long videos are generated in sequence.

Researchers can train the system on other datasets and other robot configurations as long as they have time-aligned video and control data in the right format. Nvidia noted that when connecting a control device, you must use the same control conventions and value ranges used during training. The company released the code and models along with a guide covering everything from data preparation to fine-tuning.

Nvidia cautioned that video looking realistic does not mean the physical motion of tissue is accurate. As follow-up work, it pointed to extending the scope to full surgical procedures, combining generated video with physics computation, and having surgeons evaluate the results. The company stated that the system is a research simulator, not a product validated for patient care. In effect, it widens the setting researchers have for studying instrument–tissue interaction using recordings of real surgery.