Where a model looks is not where people look: gaze heatmaps as a benchmark show Mamba localizes chest X-rays well but misses human intensity

Under one set of alignment measures, the three vision backbones show different attention shapes: convolutional networks produce more spatially diffuse saliency, vision transformers concentrate on discriminative detail, and Vision Mamba splits between natural and medical images. Earlier cross-architecture explainability comparisons relied mostly on qualitative expert inspection of heatmaps, or used gaze data as a training supervision signal, so they lacked a quantitative reference independent of training. Saliency maps from GradCAM, ScoreCAM, AblationCAM, ViTLRP and MambaLRP are compared with eye-tracker fixation-density maps on fine-grained bird classification CUB-200-2011, chest radiographs MIMIC-CXR, and facial expressions KDEF-FGD.

Alignment is split into three complementary estimators: Jensen-Shannon similarity for whether the overall distributions overlap, Chi-square similarity for discrepancy in saliency mass, and Pearson correlation for whether the mean-removed spatial structure co-varies. Prior reporting often gives a single similarity number, which cannot separate looking in the right place from looking with the right intensity. After both attribution maps and human fixation heatmaps are normalized onto the probability simplex, MambaLRP on chest X-rays attains near-maximal Jensen-Shannon and Chi-square similarity together with a Pearson correlation near zero or negative.

Freezing the backbone changes explanation quality: on the chest X-ray test set, the frozen ViT-base-F scores higher in Jensen-Shannon similarity than the unfrozen ViT-base, while its classification accuracy is lower. This comparison changes only the set of parameters updated during training, isolating backbone freezing as a controlled intervention. The two ViT variants are compared on the same data with the same ViTLRP explanation method, and alignment and accuracy move in opposite directions.

The evaluation addresses researchers and practitioners who need to judge whether model attention matches human attention, covering fine-grained natural image classification, chest radiograph diagnosis and facial expression recognition, with ResNet-50, ViT-Base and ViM-Base backbones, and it releases code and all heatmaps. Applying the conclusions to a new domain or backbone requires collecting a matching gaze benchmark and computing it through the same normalization pipeline.

The authors note that the three datasets differ in eye-tracking device, sampling frequency and observer population, so fixation-density maps are comparable within a dataset but not across datasets. CUB observers had no ornithological knowledge, and their fixation precision depends on how similar the compared bird species are; KDEF-FGD gaze data come mainly from young Western adults aged 19 to 24, and the authors note that culture and age influence facial attention patterns. ViM-base with MambaLRP shifts between the training and test splits on chest X-rays, which the authors attribute more plausibly to the numerical conditioning of the conservation-preserving propagation rule on the selective recurrence than to classifier overfitting.