Saliency maps for medical image classifiers stop guessing from gradients: unsupervised training lets a frozen classifier check the mask
Synopsis
Saliency maps for convolutional classifiers on medical images now have a new training route: SalientNet needs no annotated saliency ground truth and instead requires the generated mask to keep enough information for a frozen classifier to reproduce its original prediction within a controlled tolerance. Across BreakHis, Brain MRI ND5 and FracAtlas, it reaches higher faithfulness (Win %) and lower prediction Drop (%) than Grad-CAM, Grad-CAM++ and LIME in nearly all dataset-architecture combinations, and on FracAtlas its maps correlate spatially with practitioner-annotated fracture regions, with a mean ground-truth active fraction of up to 0.96 for ResNet-18.
Interpretation
The heat-map explanation a convolutional network gives on a medical image can now be produced by a separately trained network rather than inferred only from the explained model's gradients or perturbations. The common prior route was gradient- or perturbation-based saliency such as Grad-CAM, Grad-CAM++ and LIME, which are described as weakly sensitive to input perturbations and lacking causal grounding and mechanistic insight. On BreakHis histopathology, Brain MRI ND5 and FracAtlas musculoskeletal X-rays, with ResNet-18, MobileNetV3-Small and EfficientNet-B0 as the interpreted classifiers, SalientNet achieves higher faithfulness (Win %) and lower prediction Drop (%) across nearly all dataset-architecture combinations while keeping spurious confidence increases (Inc %) comparably low.
Generating the explanation becomes a trainable objective in which the mask must be just sufficient, neither hiding everything nor hiding nothing. Training needs no ground-truth saliency annotations; an Interpretation Loss instead requires the attribution map to retain enough information for a frozen classifier to reproduce its original prediction within a controlled dissimilarity temperature. SalientNet trains two components jointly: a CNN examiner whose variable channel convolution module extracts architecture-independent information from the explained network's intermediate feature maps, and an attribution map generator built from two coupled U-Nets that combines that information with the input image into a saliency map.
The spatial correspondence between an explanation and clinically annotated regions can be checked directly rather than judged by appearance. On the FracAtlas musculoskeletal X-ray data, explanations were compared spatially against fracture regions annotated by medical practitioners. With ResNet-18 as the interpreted classifier, the mean ground-truth active fraction reaches up to 0.96, and a mechanistic analysis shows nearly all convolutional layers contribute meaningfully to SalientNet's explanations.
Perspective
The demonstrated scope is three datasets, BreakHis histopathology, Brain MRI ND5 and FracAtlas musculoskeletal X-rays, with ResNet-18, MobileNetV3-Small and EfficientNet-B0 as the interpreted classifiers, aimed at researchers and engineers who need local validation of medical imaging models.
The faithfulness and Drop advantage is stated for nearly all dataset-architecture combinations rather than all of them; the spatial correlation with practitioner annotations is reported only on FracAtlas, and the 0.96 mean ground-truth active fraction belongs specifically to ResNet-18.