Attribution maps stop guessing: optimizing insertion and deletion curves directly, Ra-NEM tops average faithfulness across four image models
Synopsis
Attribution maps for explainable AI were previously produced by heuristics, with faithfulness only measured afterward by insertion and deletion curves. Ra-NEM turns the area under those curves into a differentiable objective, using the Gumbel top-k trick for differentiable top-k feature selection inside the neural explanation mask framework. On ImageNet-pretrained ResNet50, ConvNeXt, VGG16 and ViT it reaches the highest average faithfulness and the fastest inference, though it ranks fifth on VGG16.
Interpretation
Attribution maps can now be generated with faithfulness as a training target rather than only scored by it afterward. Earlier attribution methods came from heuristic rules, and insertion and deletion curves served only for evaluation, with no method directly optimizing their area. On 1000 images sampled from the ImageNet validation split, Ra-NEM achieved the highest average faithfulness among nine methods, and its faithfulness differences from the others were statistically significant under a two-sided paired Wilcoxon test.
The route is to rewrite the insertion curve as a top-k feature selection problem and make that selection differentiable. Optimizing the curve area directly is non-differentiable, while the usual Gumbel softmax draws one feature at a time, becoming costly for large k. The Gumbel top-k trick perturbs unnormalized log-probabilities with standard Gumbel noise and takes the top k, yielding an ordered sample without replacement in one pass that scales to high-dimensional inputs; combined with NEM-U it forms Ra-NEM, trained with six sampled k values per input.
The trained explanation module needs only a single forward pass, which keeps latency low. Other occlusion-based methods usually run a separate optimization for each new input to produce an explanation. On an RTX4090, Ra-NEM was the fastest of the compared methods with NEMt close behind; Ra-NEM ranked first in faithfulness on ResNet50, ConvNeXt and ViT, and fifth on VGG16.
Perspective
The report is aimed at researchers and engineers who need attribution maps for image classifiers. The demonstrated scope is ImageNet-pretrained ResNet50, ConvNeXt, VGG16 and ViT, evaluated on 1000 validation images, with robustness and randomization metrics on a 100-image subset. The method itself only requires a differentiable explained model, and binary masking makes it in principle extensible to text or graph modalities, but that extension has not been tested experimentally.
Faithfulness scores are sensitive to the perturbation scheme, differing across noise, zero and blur perturbations, though the method ranking stays stable across models. Ra-NEM is a trained method, and faithfulness varied by at most 0.021 across repeated training runs. Larger sampled k during training improves faithfulness at the cost of longer training, a trade-off. On VGG16, Ra-NEM trails the top method by 0.139, which the authors partly attribute to not optimizing directly on the output when generating attributions.