Multimodal explanations without model internals: MultiSHAP scores each image-patch and text-token pair
Synopsis
Multimodal models answer questions and retrieve images, yet which image region and which word jointly push a given output score has been hard to quantify. MultiSHAP applies the Shapley Interaction Index from cooperative game theory to estimate, for every image-patch and text-token pair, a second-order synergy or suppression score, using only a query interface that allows separate perturbation of image and text and returns a task-relevant scalar. On 500 VQAv2 samples, deleting patches in the resulting ranking lowered the target-answer probability by 0.1720 on average, versus 0.0032 for the lowest-ranked patches.
Interpretation
A multimodal model's output can now be attributed to specific image-patch and text-token pairings, not only to a whole image or a whole text modality. Attention maps and Grad-CAM give visual or architectural attribution signals but do not explicitly quantify second-order interactions between individual patches and tokens, and they require gradients, activations or attention weights. On 500 VQAv2 samples, deleting patches in MultiSHAP's element ranking lowered the target-answer probability by 0.1720±0.0102 on average, versus 0.0032±0.0022 for the lowest-ranked patches; deletion AUC was 0.4105±0.0024 against 0.5005 for a random ordering.
The method treats each image patch and each text token as players in a cooperative game and defines their interaction from the score difference between joint and separate presence. The Shapley Interaction Index measures how much a pair's joint presence changes the output beyond the sum of its individual contributions, with positive values indicating synergy and negative values suppression. In the VQAv2 visual question answering setting and the Flickr30K image-text retrieval setting, using ViLT-VQA and fine-tuned CLIP ViT-B/32 checkpoints respectively, excluded patches are filled with mid-gray 0.5 and excluded tokens are deleted, with 128 accepted coalitions per pair.
Pair-level tests give partial support: the estimated interactions carry information beyond the product of two unimodal attributions, but do not beat a saliency-only null under every criterion. In a partner-swap test that holds a patch fixed and exchanges only its paired token, the pair favored by MultiSHAP produced the larger prediction change in 71.0% of cases, and 73.2% after correcting for marginal effects. On Flickr30K-Entities region-phrase annotations, the saliency-only null ranked all candidate correspondences better overall (AUC 0.653 versus 0.612), while MultiSHAP's single most confident pair was more often an annotated correspondence (pointing game 0.630 versus 0.470).
Perspective
The demonstrated scope is query-accessible research models: ViLT-based models on VQAv2 and GMDB, and CLIP ViT-B/32 on MSCOCO and Flickr30K. The method requires an interface that permits separate perturbation of image and text and returns a task-relevant scalar score, either directly or through a ranked output from which such a score can be deterministically constructed. In the medical case the text is the same fixed diagnostic question for every patient, with no individualized phenotype description, so that case examines interactions between facial regions and tokens of a fixed prompt. Evaluation on a commercial multimodal API remains future work.
The authors describe the pair-level evidence as partial: under effect-magnitude and region-phrase correspondence criteria, MultiSHAP does not rank all candidate pairs better than a saliency-only null. Explanations are materially sensitive to the masking operator, with zero fill, Gaussian blur and [UNK] replacement changing attribution rankings by more than reseeding the estimator at the same budget. The ordering of MSR and SDR across three rare-disease cohorts is exploratory, with no inferential test and cohorts unmatched on sample size, age, ancestry and acquisition conditions.