SurgTAP-3D turns 520K uncalibrated surgical video frames into 1.4M 3D point trajectories
Synopsis
Vision-language-action models and world models for minimally invasive surgery need large-scale 3D supervision, yet public surgical datasets are largely ex-vivo or simulated and annotate either tissue or instruments, leaving tool-tissue contact regions unmarked. SurgTAP-3D converts uncalibrated monocular surgical videos into pseudo-scale camera poses, depth and 2D/3D point trajectories through contact-region prompting, active-tool segmentation, dynamic bundle adjustment and global 4D refinement, gathering 4,549 action clips, 519,642 frames and about 1.38 million trajectories across robotic and manual procedures. On the SurgT, SurgPose and STIR benchmarks, the pipeline reaches an average AJ3D of 56.57 for 3D tracking, above the compared methods in the table.
Interpretation
Public surgical motion datasets have mostly been captured ex vivo or in simulation, often cover a single procedure or acquisition mode, and annotate either tissue or instruments, so tool-tissue contact regions are not marked separately. SurgTAP-3D extends the source to in-vivo surgical video, includes both robotic and manual procedures, and maps cholecystectomy, prostatectomy, gastric bypass, hysterectomy and rectal resection onto five unified action categories. The dataset aggregates 4,549 clips, 519,642 frames and 1,376,987 trajectories, with camera intrinsics, poses, depth, instrument masks and 2D and 3D point trajectories attached to each clip.
The annotations are not drawn frame by frame by hand; a pipeline chains foundation-model priors with geometric optimization to generate them automatically. The pipeline first uses SAM2 with a first-frame contact-region box and positive and negative point prompts to obtain active-instrument masks, TAPNext for 2D point tracking, and Endo3R to estimate depth, camera poses and intrinsics. Dynamic bundle adjustment then takes these predictions as initialization and jointly refines keyframe poses, intrinsics and depth, while global 4D refinement corrects trajectories with optical-flow consistency, instrument surface constraints, articulated ARAP regularization and temporal smoothness.
On public benchmarks, the pipeline's tracking and depth results exceed the compared methods listed in the tables. The comparison spans three families: RGB-only 2D trackers, RGBD methods tracking in camera coordinates, and RGBD-plus-pose methods tracking in global coordinates. Average 3D tracking over SurgT, SurgPose and STIR is AJ3D 56.57 and APD3D 64.77, against 55.69 average AJ3D for the next-highest listed method, TAPNext; depth Abs Rel falls from 0.124 for Endo3R to 0.103 on SCARED and from 0.170 to 0.156 on Hamlyn.
Perspective
The demonstrated setting is monocular in-vivo minimally invasive surgery video covering robotic and manual procedures and five unified action categories, with pseudo-scale geometry; the authors note that metric reasoning in the robot workspace would require additional calibration.
The authors state that geometry recovered from monocular video is only sequence-consistent and pseudo-scale, and that trajectories may disappear, drift or become temporally unstable under long-term instrument disappearance, severe occlusion and fast motion, more often in tissue retraction, needle grasping and coagulation. Annotation quality is measured by four internal consistency metrics—survival ratio, instrument mask consistency, temporal smoothness ratio and tracking confidence ratio—which the authors explicitly describe as indicators rather than direct measures of geometric accuracy.