Brain MRI foundation models lose accuracy on a new dataset: not because they remember the source, but because each dataset encodes the sequence along different directions

Frozen brain MRI features lose the ability to separate imaging sequences on a new dataset, even though they separate T1, T1c, T2 and FLAIR reliably inside a single dataset. This cross-dataset failure is usually blamed on the features encoding which dataset each scan came from, and the source dataset is indeed decodable from brain MRI. Across 17 public datasets and eight frozen representations, within-dataset sequence balanced accuracy is 0.84 to 0.97, but a probe trained on 16 datasets and tested on the seventeenth drops to 0.44 to 0.55 for the baseline encoders and both brain-specific models, while 3DINO and SAM-Med3D keep 0.72 and 0.69.

The failure is not a whole-dataset shift but a disagreement between the directions each dataset uses to separate the classes. If each dataset were merely displaced as a whole, subtracting each dataset's mean would recover transfer. Neither centring test changes sequence transfer significantly for any representation, and most lose slightly, whereas higher direction consistency goes with better transfer, with a Spearman correlation of 0.66 across all representations and held-out datasets.

Erasing the dataset signal afterwards with INLP, LEACE, CORAL or adversarial training does not improve sequence transfer, because those directions already contain the per-dataset sequence differences. The methods do remove the dataset signal inside the training datasets, lowering dataset decodability from 0.84 to 0.95 down to 0.03 to 0.26, and CORAL to at most 0.11. None of the 32 method-representation combinations improves sequence transfer significantly, almost all lower it and seven do so significantly, and after LEACE sequence decodability falls by 0.14 to 0.24 for every network representation while removing 15 random directions costs only 0.01 to 0.05.

Only models pretrained on broad multi-organ collections encode the sequence consistently enough to transfer across datasets. An earlier brain MRI control found no reliable advantage of pretraining over an untrained network, but tested only one architecture family. 3DINO and SAM-Med3D exceed the untrained CNN on all ten held-out datasets, while the brain-specific BrainIAC and BrainFM show no significant advantage over it, and a randomly weighted 3DINO reaches only 0.50, like the baseline encoders.

The diagnostic is aimed at researchers using frozen encoders on pooled public brain MRI data: the sequence is the only content label shared by all 17 datasets, and sex serves as a second anatomical label in ten datasets. Direction consistency is computed from class means alone, so before deployment the pool consistency of the training datasets ranks representations, and on a new unlabelled dataset cluster consistency warns which datasets a chosen encoder will fail on; the centring test, run on labelled datasets, shows whether aligning datasets afterwards is worth trying. If a few target labels are available, the probe's accuracy on them is the more direct check.

The sequence is a global and comparatively easy property, sex covers only ten datasets with weaker effects, and only LEACE and CORAL were applied as removal methods for it. The label-free failure prediction assumes the number of sequences in a new dataset is known and reaches only a moderate correlation. Transfer is measured with linear probes on frozen features, and non-linear probes or fine-tuning may behave differently. The paired tests use ten multi-sequence held-out datasets, which limits statistical power, and the pool-consistency ranking rests on 11 points. Each encoder receives the input scaling of its own pretraining, so part of the differences between models may reflect preprocessing. Pretraining data of the foundation models are not fully documented, so the overlap control can only exclude datasets known or likely to be included.