# Vision Foundation Models

5 related articles

Subscribe to updates

arXiv Nov 5, 2026
Surgical vision encoders stop starting from natural images: InnerLogic-Surg leads all 20 surgical tasks with a frozen backbone
Surgical image analysis has long relied on encoders pretrained on natural images or on specialist models trained from scratch on comparatively small annotated datasets. InnerLogic-Surg starts from DINOv3 and applies continual self-supervised pretraining to millions of frames drawn from 11,965 public surgical videos, about 2,900 hours across more than fifteen procedure types, yielding an encoder family of 21M to 300M parameters. Under a frozen-backbone protocol with only a linear readout trained, the family achieves the best linear-probe performance on all 20 tasks spanning workflow, instrument and anatomy recognition, segmentation, detection and monocular depth, with ViT-L leading on 18 axes and ViT-B best on GraSP step recognition and monocular depth.
Surgical image analysis has long relied on encoders pretrained on natural images or on specialist models trained from scratch on comparatively small annotated datasets. InnerLogic-Surg starts from DINOv3 and applies continual self-supervised pretraining to millions of frames drawn from 11,965 public surgical videos, about 2,900 hours across more than fifteen procedure types, yielding an encoder family of 21M to 300M parameters. Under a frozen-backbone protocol with only a linear readout trained, the family achieves the best linear-probe performance on all 20 tasks spanning workflow, instrument and anatomy recognition, segmentation, detection and monocular depth, with ViT-L leading on 18 axes and ViT-B best on GraSP step recognition and monocular depth.
Kun Yuan（First author）
InnerLogic-Surg Continual Pretraining Vision Foundation Models Self-Supervised Learning Depth Estimation
arXiv Oct 5, 2026
OmniAct3D adapts perspective-trained vision foundation detectors to panoramic ERP, gaining 2.96 NDS over the previous best 3D detector on Spheriverse
The work proposes OmniAct3D, a framework that adapts perspective-trained Vision Foundation Model detectors to equirectangular projection (ERP) panoramic images by modeling spherical viewing rays and periodic spatial structure with an ERP-Ray Geometry Adapter, grounding each hypothesis in panoramic evidence and converting it into a structured geometric action via a Visual-Action Reasoning Chain, and re-encoding object regions at higher resolution for heading estimation with an Appearance-Guided Heading Expert; experiments report a 2.96 NDS improvement over the previous best 3D detector on Spheriverse, a 24.87 mAP improvement over the unadapted VFM baseline on PanoMMOcc, and 95-98% retention of same-configuration mAP by VARC under target-specific geometry adaptation.
The work proposes OmniAct3D, a framework that adapts perspective-trained Vision Foundation Model detectors to equirectangular projection (ERP) panoramic images by modeling spherical viewing rays and periodic spatial structure with an ERP-Ray Geometry Adapter, grounding each hypothesis in panoramic evidence and converting it into a structured geometric action via a Visual-Action Reasoning Chain, and re-encoding object regions at higher resolution for heading estimation with an Appearance-Guided Heading Expert; experiments report a 2.96 NDS improvement over the previous best 3D detector on Spheriverse, a 24.87 mAP improvement over the unadapted VFM baseline on PanoMMOcc, and 95-98% retention of same-configuration mAP by VARC under target-specific geometry adaptation.
Runtong Wu（First author）
OmniAct3D Panoramic 3D Detection Vision Foundation Models Visual-Action Reasoning Chain ERP-Ray Geometry Adapter
arXiv Oct 2, 2026
Latent-Foresight jointly trains a tokenizer and a flow-matching predictor, consistently beating two-stage baselines on future scene understanding in Cityscapes and nuScenes
The work proposes Latent-Foresight, an end-to-end framework that jointly learns a VFM feature tokenizer and a flow-based generative dynamics model, using stop-gradient, latent normalization, a denoised-latent reconstruction loss and a noise schedule to prevent latent collapse, and it consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons on Cityscapes, nuScenes and Kubric while eliminating separate training stages, including during high-resolution adaptation.
The work proposes Latent-Foresight, an end-to-end framework that jointly learns a VFM feature tokenizer and a flow-based generative dynamics model, using stop-gradient, latent normalization, a denoised-latent reconstruction loss and a noise schedule to prevent latent collapse, and it consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons on Cityscapes, nuScenes and Kubric while eliminating separate training stages, including during high-resolution adaptation.
Efstathios Karypidis（First author）
Latent-Foresight Flow Matching World Model Latent Tokenizer Vision Foundation Models
arXiv Sep 25, 2026
Sparse autoencoders trained on 909,873 CT/MRI slices decompose medical foundation-model embeddings into language-describable concepts, recovering 87.8% of downstream performance with 10 features
Trained on 909,873 2D CT and MRI slices from the TotalSegmentator dataset, this study fits Matryoshka sparse autoencoders with BatchTopK sparsification on frozen embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) foundation models alongside a random-weight baseline, finding that sparse features reconstruct original embeddings with R2 up to 0.941, recover 87.8% of downstream performance with only 10 features (99.4% dimensionality reduction), preserve 97.7% of dense retrieval quality with five-feature fingerprints, correspond to monosemantic concepts expressible in language as verified by an independent LLM judge, and enable zero-shot language-driven image retrieval on a single clinical text query.
Trained on 909,873 2D CT and MRI slices from the TotalSegmentator dataset, this study fits Matryoshka sparse autoencoders with BatchTopK sparsification on frozen embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) foundation models alongside a random-weight baseline, finding that sparse features reconstruct original embeddings with R2 up to 0.941, recover 87.8% of downstream performance with only 10 features (99.4% dimensionality reduction), preserve 97.7% of dense retrieval quality with five-feature fingerprints, correspond to monosemantic concepts expressible in language as verified by an independent LLM judge, and enable zero-shot language-driven image retrieval on a single clinical text query.
Stanford University Technical University of Munich Artificial Intelligence in Medicine (Canada)
Philipp Wesp（First author, Corresponding author）
Sparse Autoencoder Matryoshka Sparse Autoencoder Medical Image Retrieval Mechanistic Interpretability Vision Foundation Models
arXiv Sep 26, 2026
Frozen DINOv3 features injected into a 3D U-Net via Room-Lite mixing and calibrated fusion reach Dice 0.758 for 3DRA aneurysm segmentation and remove all cross-dataset failures
The work proposes DINO-3DRA, a dual-path framework that injects frozen 2D vision foundation model DINOv3-Small features into a trainable 3D U-Net backbone through Room-Lite spatial mixing and calibrated residual fusion, achieving state-of-the-art aneurysm segmentation on multi-centre 3D rotational angiography (3DRA) @neurIST data (233 patients, four institutions) with Dice 0.758, HD95 2.75 mm and a 13% gain over nnU-Net using only 5.72M trainable parameters; ablations attribute the gains to structured cross-dimensional transfer rather than loss design, and without fine-tuning on CADA (n=45) and SHINY-ICARUS (n=30) it reduces Dice<0.5 failures from 11.1% and 3.3% to 0%.
The work proposes DINO-3DRA, a dual-path framework that injects frozen 2D vision foundation model DINOv3-Small features into a trainable 3D U-Net backbone through Room-Lite spatial mixing and calibrated residual fusion, achieving state-of-the-art aneurysm segmentation on multi-centre 3D rotational angiography (3DRA) @neurIST data (233 patients, four institutions) with Dice 0.758, HD95 2.75 mm and a 13% gain over nnU-Net using only 5.72M trainable parameters; ablations attribute the gains to structured cross-dimensional transfer rather than loss design, and without fine-tuning on CADA (n=45) and SHINY-ICARUS (n=30) it reduces Dice<0.5 failures from 11.1% and 3.3% to 0%.
University of Manchester Manchester Academic Health Science Centre
Jiayang Lu（First author）Ali Sarrami‐Foroushani（Corresponding author）
DINO-3DRA Cerebral Aneurysm Segmentation Cross-Dimensional Feature Transfer Vision Foundation Models U-Net

You’re all caught up

Surgical vision encoders stop starting from natural images: InnerLogic-Surg leads all 20 surgical tasks with a frozen backbone

OmniAct3D adapts perspective-trained vision foundation detectors to panoramic ERP, gaining 2.96 NDS over the previous best 3D detector on Spheriverse

Latent-Foresight jointly trains a tokenizer and a flow-matching predictor, consistently beating two-stage baselines on future scene understanding in Cityscapes and nuScenes

Sparse autoencoders trained on 909,873 CT/MRI slices decompose medical foundation-model embeddings into language-describable concepts, recovering 87.8% of downstream performance with 10 features

Frozen DINOv3 features injected into a 3D U-Net via Room-Lite mixing and calibrated fusion reach Dice 0.758 for 3DRA aneurysm segmentation and remove all cross-dataset failures