Medical transfer stops leaning on layout: moving global attention to the feature axis turns the same pretraining into roughly three times the gain

In medical backbones, which axis carries global attention changes how well pretrained knowledge transfers. Moving global mixing from spatial tokens to channel tokens markedly strengthens downstream gains from the same ImageNet pretraining while leaving from-scratch performance essentially unchanged. Transferred encoders have inherited the standard vision transformer, where global modeling is dominated by self-attention over spatial tokens and channel interaction is left to a mostly static feed-forward network; channel-aware hybrids add channel attention onto a spatially mixed backbone, leaving the two axes coupled. Under matched capacity, the feature-axis backbone converts pretraining into a breast-ultrasound Dice gain of +15.1 against +4.3 for a spatial ViT-S, roughly a threefold difference; scaling the spatial baseline to ViT-B (90.6M) or using a pretrained ResNet50 still yields only +5.7 and +6.0.

The design assigns spatial extraction and global mixing to two disjoint operators. Depthwise convolution refines local structure per channel without combining channels, and global interaction runs as single-head self-attention along the feature axis, where each channel token is a whole-image response map, so attention reweights whole-image channel responses rather than spatial locations. Existing channel-aware designs keep pointwise projections that linearly mix channels before feature attention, and spatial mixing persists throughout, so pretraining cannot be attributed to one axis. Evaluation spans 23 medical datasets and tasks under one protocol with an identical U-Net decoder for segmentation; the feature axis takes the best Dice on ten of eleven held-out segmentation tasks, mean 84.3 against 79.8 for ViT-UNet and 77.4 for DINOv2-UNet, at 23.1M parameters versus 90.6M.

The report covers classification, segmentation and low-data linear probing across 23 medical datasets and tasks, with an identical U-Net decoder for segmentation, end-to-end fine-tuning for classification, and a frozen backbone with a linear head for probing. The authors propose stronger pipelines such as nnU-Net, other architecture families and self-supervised pretraining as next steps.

The authors read centered kernel alignment as a corroborative correlate of the measured transfer gap, not direct proof of feature reuse. Low-data linear probing is dataset-dependent: the feature axis leads on PathMNIST, OCTMNIST, PneumoniaMNIST and BloodMNIST, while DINOv2 and ViT-S/16 lead on a few others. From scratch, the axis swap costs only about 1 Dice, within seed variance, so the axis does not make the network intrinsically stronger.