Medical transfer stops leaning on layout: moving global attention to the feature axis lifts ultrasound Dice from 53.1 to 77.7

The axis of global attention is itself a lever for transfer design: placing global mixing on channel tokens rather than spatial tokens lets the same natural-image pretraining carry more usable structure into medical tasks. Earlier channel-aware backbones added channel attention to a backbone still dominated by spatial mixing, leaving the two axes entangled so the feature axis could not be measured alone; standard ViTs put all global modeling on spatial self-attention. Under matched tokenization, capacity and protocol, the feature and spatial axes perform essentially the same from scratch, while identical ImageNet pretraining yields a +15.1 Dice transfer gain on breast ultrasound for the feature axis against +4.3 for a capacity-matched spatial baseline, roughly a threefold difference.

The gain comes from the feature axis itself, not from a larger parameter budget or a stronger source task. Scaling the spatial baseline to ViT-B or using a pretrained convolutional encoder still yields only +5.7 and +6.0 Dice gaps, far below the feature axis's +15.1. The three backbones have comparable ImageNet-1K top-1 accuracy, with the feature axis ahead by only a few points, too little to explain Dice-level differences; a strict in-place axis swap on ultrasound, abdominal CT and cardiac MRI shows the feature axis converting pretraining into a larger gain on all three modalities.

Feature-axis attention shapes pretraining toward channel-correlation statistics, texture, intensity and co-activation, which recur across scanners and anatomy, whereas spatial-layout priors are largely invalidated by medical domain shift. Spatial-token attention instead learns object-layout priors specific to natural scenes, which are unstable in medical images. Centered kernel alignment between pretrained and from-scratch encoders stays higher at early and mid depth for the feature axis and collapses only at the last stage, while spatial-axis alignment decays monotonically; the authors read this as a corroborative correlate of the transfer gap rather than direct proof of feature reuse.

The conclusion rests on 23 medical datasets and tasks, covering MedMNIST-2D classification, BUSI and dataset B breast ultrasound, BUSI plus 10 MSD-derived 2D segmentation tasks, and linear probing on 8 MedMNIST datasets at 1%, 10% and 100% labels. Segmentation experiments attach an identical U-Net decoder across encoders, classification is fine-tuned end-to-end, and linear probing freezes the backbone. The authors propose next steps of stronger pipelines such as nnU-Net, other architecture families and self-supervised pretraining.

From scratch, the axis swap costs only about 1 Dice and removing the spatial tail about 1 Dice, within seed variance, so the axis does not make the network intrinsically stronger and matters only after pretraining. Low-data linear probing is dataset-dependent: the feature axis leads on PathMNIST, OCTMNIST, PneumoniaMNIST and BloodMNIST but trails DINOv2 and ViT-S/16 on a few others. Grouping by modality shows the largest gain on ultrasound, less on MRI and least on high-contrast CT, with the near-saturated colon CT task marginally favoring DINOv2. The authors position the CKA alignment as a corroborative correlate of the transfer gap, not direct proof of feature reuse.