An MRI foundation model aligns by anatomy: no prompt box, zero-shot 3D organ localization matches MedSAM2

MRI foundation models are starting to align images and text at the level of individual anatomical structures rather than whole images, and to separate normal from abnormal appearances of the same organ in embedding space. Earlier MRI vision-language models learned image-level representations, and even when aligned with radiology reports they lacked explicit anatomy-level structure, making it hard to tell whether a given organ or region is normal or abnormal. On 2000 held-out studies with reports, zero-shot cross-modal retrieval reaches 90.13 mAP for entire-report image-to-text, above Decipher-MR at 77.80, BiomedCLIP at 70.90 and MedImageInsight at 71.70.

The model uses weakly annotated anatomy boxes and health status extracted from reports as supervision, placing localization, image-text alignment and normal-versus-abnormal separation in one contrastive objective. Anatomy boxes come from an open-source segmentation model, proprietary segmentation models and DICOM header coordinate transfer, while anatomies and health status are extracted from reports by a large language model, with only a subset of processed reports manually reviewed. Pretraining covers about 50k studies and 491k series across 16 body regions, with reports for 25.5k studies; the vision encoder is a 3D ViT-Base and the text encoder is initialized from PubMedBERT.

With the encoder frozen, a lightweight segmentation decoder transfers the pretrained representations to dense prediction and holds accuracy when labels are scarce. End-to-end task-specific models typically require full encoder fine-tuning and degrade sharply under low label budgets. Knee segmentation averages 87.3 Dice, above MedNeXt and VNet at 85.9 and nnUNet3D at 81.4; with only 10 labeled cases it reaches 68.2 Dice while VNet is near zero.

The demonstrated scope is 3D MRI: pretraining on about 50k studies and 491k series across 16 body regions, plus downstream tasks in retrieval, abnormality detection, segmentation, 3D organ localization and structured prostate report generation. Localization is evaluated on the AMOS22 MR validation set, 247 image-organ pairs after excluding 12 pairs with unreliable ground-truth boxes, using 3D intersection-over-union between predicted and ground-truth boxes. Segmentation uses a frozen encoder with a SegResNet decoder on AMOS, an internal pelvis dataset, CirrMRI600+ and knee data. Report generation is on an internal prostate dataset with 250 training and 50 validation studies and no separate test set.

Pretraining supervision is weak and noisy: anatomy boxes come from segmentation models and DICOM coordinate transfer, health status is extracted from reports by a large language model, only a subset of reports was manually reviewed, and weak masks are reduced to bounding boxes. The encoder uses 8-voxel patches, so structures smaller than one patch along any axis cannot be resolved, and zero-shot localization still trails multi-slice prompted MedSAM2 and dataset-specific supervised models on small organs such as the adrenal glands. Abnormality is modeled as a binary healthy or abnormal state per organ, propagated to all mentions of that organ within a report, without severity, co-occurring findings or lesion-level detail, and abnormality evaluation focuses mainly on a single organ and disease. The localization decoder detects a fixed anatomy class list, so anatomies outside that vocabulary cannot be localized.