Surgical vision encoders stop starting from natural images: InnerLogic-Surg leads all 20 surgical tasks with a frozen backbone

Surgical vision analysis can now use a general encoder continually pretrained on surgical video instead of training a separate model for each task. Earlier practice either reused encoders pretrained on natural images or trained specialist models from scratch on comparatively small datasets, so extending to new tasks, procedures or institutions often required additional annotation and retraining. Under a frozen-backbone evaluation that trains only a linear layer, the InnerLogic-Surg family achieves the best linear-probe result on all 20 surgical tasks, with ViT-L leading on 18 axes, ViT-B best on GraSP step recognition and monocular depth, and the two variants tied on PitVis step recognition.

Continual pretraining keeps general visual priors while pushing the representation toward surgical scenes. Continued self-supervised training from a general foundation model can erode the spatial structure already present in the pretrained representation, creating tension between surgical specificity and broad visual priors. Training initializes from DINOv3 and jointly optimizes the DINO image-level loss and the iBOT masked patch-level loss on 3.6 million decontaminated frames, with a second stage adding a Gram-anchoring constraint that keeps pairwise patch-feature similarities close to those of an earlier checkpoint.

High-resolution dense prediction comes from a separately trained feature upsampler rather than from enlarging the backbone input. Running the backbone at resolutions beyond its pretraining crop size degrades the patch-level feature map and produces grid artifacts. AnyUp lifts the frozen backbone's native-resolution patch tokens into dense features; compared at several scales on the same frames, anatomical and instrument contours stay crisp without extra backbone passes.

Evaluation is limited to a frozen-backbone plus linear-readout protocol covering 20 image tasks and a separately reported SurgicalActions160 clip-recognition result; the pretraining corpus consists of internet-sourced public surgical videos and was decontaminated against the evaluation test sets.

On SurgicalActions160, InnerLogic-Surg ViT-L reaches 53.1% top-1, comparable to LemonFM at 53.1, below DINOv3 ViT-L at 56.3, and 6.3 points behind the TimeSformer-B video model SurgLaVi/SurgCLIP at 59.4; the downstream contribution of the Gram-anchoring stage is not individually ablated; in the ablation statistics, six of fourteen tasks show differences too small to reliably distinguish configurations, and frame correlation within a video may underestimate true uncertainty.