Brain MRI foundation model drops the single-sequence habit: 1.55M scans, up to 15% few-shot gain on gliosis and MS

Neuroimaging foundation models mostly handled a single MRI sequence, and multimodal methods were typically evaluated only when all sequences were present, so it was unclear whether joint encoding truly beats the best single-sequence model. Prior work either modeled one sequence or reported the all-sequences-present case, leaving no controlled unimodal-versus-multimodal comparison to separate multimodal benefit from simply having more data at test time. Across 47 tasks in three health systems and 12 public datasets, Neuro-JEPA beat BrainIAC, VoCo and NeuroVFM in most comparisons, with mean AUROC gains of 4.4–6.4% and AUPRC gains of 6.4–9.4% on public datasets, despite pretraining on fewer scans than NeuroVFM.

The model puts all three contrasts into one network and learns by predicting latent representations of masked regions rather than reconstructing pixels. It builds on the Joint-Embedding Predictive Architecture, in which an online encoder predicts latent targets for masked regions and a momentum encoder supplies those targets, avoiding the cost of 3D image interpolation. Pretraining used 1,551,862 T1w, T2w and FLAIR scans from 282,693 patients acquired between 2009 and 2025; the ViT-Base variant activates 86 million of 122 million parameters, comparable to the baselines.

Three design changes make JEPA work on neuroimaging: multiscale masking, sparse expert routing, and down-weighting background voxels in the loss. Applying the video-domain V-JEPA 2 configuration directly performed poorly on brain MRI, because aggressive masking along the depth axis hides the high-frequency detail needed for small lesions, and roughly 40–60% of skull-stripped voxels are background. Stepwise ablation showed multiscale masking alone added 1.5% mean AUROC and 2.5% AUPRC, adding MoE contributed a further 3.9% and 3.4%, and the foreground-aware L1 loss added 0.9% and 2.8%.

Evaluation covers 47 tasks across three health systems and 12 public datasets, including unimodal, multimodal, few-shot, cross-cohort and unseen diffusion-weighted imaging settings. Pretraining used only T1w, T2w and FLAIR sequences, and experiments used down-sampled scans with fixed patch sizes. Model weights are released for non-commercial, non-derivative, non-clinical academic research, with code public.

The advantage was substantially larger on the internal NYU cohort than on the external MGH cohort, indicating limits of single-system pretraining; the authors call for more institutional, demographic and acquisition heterogeneity. Diffusion-weighted, susceptibility-weighted, contrast-enhanced and functional MRI, as well as CT, were not included in pretraining, and higher spatial resolution and alternative patching strategies remain for future evaluation.