# 3D Point Tracking

Vision-language-action models and world models for minimally invasive surgery need large-scale 3D supervision, yet public surgical datasets are largely ex-vivo or simulated and annotate either tissue or instruments, leaving tool-tissue contact regions unmarked. SurgTAP-3D converts uncalibrated monocular surgical videos into pseudo-scale camera poses, depth and 2D/3D point trajectories through contact-region prompting, active-tool segmentation, dynamic bundle adjustment and global 4D refinement, gathering 4,549 action clips, 519,642 frames and about 1.38 million trajectories across robotic and manual procedures. On the SurgT, SurgPose and STIR benchmarks, the pipeline reaches an average AJ3D of 56.57 for 3D tracking, above the compared methods in the table.

Vision-language-action models and world models for minimally invasive surgery need large-scale 3D supervision, yet public surgical datasets are largely ex-vivo or simulated and annotate either tissue or instruments, leaving tool-tissue contact regions unmarked. SurgTAP-3D converts uncalibrated monocular surgical videos into pseudo-scale camera poses, depth and 2D/3D point trajectories through contact-region prompting, active-tool segmentation, dynamic bundle adjustment and global 4D refinement, gathering 4,549 action clips, 519,642 frames and about 1.38 million trajectories across robotic and manual procedures. On the SurgT, SurgPose and STIR benchmarks, the pipeline reaches an average AJ3D of 56.57 for 3D tracking, above the compared methods in the table.

TrackEverything represents videos as persistent 3D scene tracks in world coordinates and, through voxelization-based de-duplication at sliding-window boundaries, an endpoint-then-trajectory decomposition, and 3D WAFT feature sampling, becomes the first 3D tracker to follow all visible points across videos exceeding 1000 frames within 40 GB of GPU memory, outperforming all open-source all-frame dense 3D trackers by more than 20% APD on TAPVid-3D short clips while remaining competitive with state-of-the-art sparse trackers on long sequences.

TrackEverything represents videos as persistent 3D scene tracks in world coordinates and, through voxelization-based de-duplication at sliding-window boundaries, an endpoint-then-trajectory decomposition, and 3D WAFT feature sampling, becomes the first 3D tracker to follow all visible points across videos exceeding 1000 frames within 40 GB of GPU memory, outperforming all open-source all-frame dense 3D trackers by more than 20% APD on TAPVid-3D short clips while remaining competitive with state-of-the-art sparse trackers on long sequences.