Skip to main content

# Continual Pretraining

2 related articles

Subscribe to updates

arXiv Nov 5, 2026
Surgical vision encoders stop starting from natural images: InnerLogic-Surg leads all 20 surgical tasks with a frozen backbone
Surgical image analysis has long relied on encoders pretrained on natural images or on specialist models trained from scratch on comparatively small annotated datasets. InnerLogic-Surg starts from DINOv3 and applies continual self-supervised pretraining to millions of frames drawn from 11,965 public surgical videos, about 2,900 hours across more than fifteen procedure types, yielding an encoder family of 21M to 300M parameters. Under a frozen-backbone protocol with only a linear readout trained, the family achieves the best linear-probe performance on all 20 tasks spanning workflow, instrument and anatomy recognition, segmentation, detection and monocular depth, with ViT-L leading on 18 axes and ViT-B best on GraSP step recognition and monocular depth.
Surgical image analysis has long relied on encoders pretrained on natural images or on specialist models trained from scratch on comparatively small annotated datasets. InnerLogic-Surg starts from DINOv3 and applies continual self-supervised pretraining to millions of frames drawn from 11,965 public surgical videos, about 2,900 hours across more than fifteen procedure types, yielding an encoder family of 21M to 300M parameters. Under a frozen-backbone protocol with only a linear readout trained, the family achieves the best linear-probe performance on all 20 tasks spanning workflow, instrument and anatomy recognition, segmentation, detection and monocular depth, with ViT-L leading on 18 axes and ViT-B best on GraSP step recognition and monocular depth.
Kun Yuan（First author）
InnerLogic-Surg Continual Pretraining Vision Foundation Models Self-Supervised Learning Depth Estimation
arXiv May 30, 2026
Interpolation-localized middle layers plus weight freezing let gemma-3-4b multilingual CPT beat the base model on average for reading comprehension
Using gemma-3-4b continual pretraining (CPT) across five language families, the authors interpolate pre- and post-CPT model states to localize forgetting, finding that middle-layer reversion gives the largest comprehension recovery while translation effects vary by family and direction; guided by this, they compare layer freezing, layer-range L2 regularization, post-hoc layer reversion, and model souping, and find that preserving the interpolation-identified layer weights substantially reduces comprehension loss relative to joint CPT, with layer freezing exceeding base model performance on average, while translation results are mixed.
Using gemma-3-4b continual pretraining (CPT) across five language families, the authors interpolate pre- and post-CPT model states to localize forgetting, finding that middle-layer reversion gives the largest comprehension recovery while translation effects vary by family and direction; guided by this, they compare layer freezing, layer-range L2 regularization, post-hoc layer reversion, and model souping, and find that preserving the interpolation-identified layer weights substantially reduces comprehension loss relative to joint CPT, with layer freezing exceeding base model performance on average, while translation results are mixed.
Northeastern University
Sanchit Ahuja（First author）
Catastrophic Forgetting Continual Pretraining Interpolation-based Localization Layer Freezing Multilingual Model Adaptation

You’re all caught up