VLM attribution stops at the whole image: MOSAIC traces influence to phrases and regions

Training-data attribution for vision-language models moves from the whole image or the whole example down to phrases and image regions inside that example. Earlier data-attribution methods treat the image as a monolithic unit and answer only which training example mattered, not which concept carried the influence or where it appeared. Across two model families, Qwen and Gemma, and three tasks, image captioning, chest X-ray understanding, and visual question answering, the learned latents preserve task performance while their visual footprint shrinks markedly.

A shared sparse latent space holds image and text representations together, so one set of feature coordinates can correspond to both a phrase and an image region. Existing multimodal sparse autoencoders yield interpretable image-text features but are not designed for training-data attribution and lack position resolution and phrase-region correspondence. The method first learns a shared TopK dictionary, then groups patches into regions with group and exclusive sparsity so visual features concentrate on coherent regions, and aligns phrases with regions using the frozen model's phrase-to-patch attention as the correspondence signal.

Latents ranked by influence are more behaviorally consequential than latents ranked by activation magnitude or firing frequency. Removing features chosen by activation magnitude, firing frequency, or random ranking perturbs predictions substantially less. On GQA with Gemma, removing the top 256 influence-ranked latents flips more predictions than activation ranking does, and the same trend holds across datasets and models.

Removing the image strongly reorders training-example attribution while latent-level rankings stay positively correlated. Previously it was not possible to separate whether visual evidence changes the training-example support or the concepts carrying the influence. Across six model-dataset settings, latent influence correlations stay positive up to a high value, while example-level correlations fall as low as a much lower value.

The work targets researchers and engineers who need to audit vision-language model predictions, in the settings of image captioning, chest X-ray understanding, and visual question answering, at a model scale of roughly two billion parameters, with all experiments run on a single GPU. The method inserts a sparse autoencoder at a chosen intermediate layer, and attribution depends on that layer's representation quality and dictionary capacity. Readers following up can watch the authors' proposed directions of multi-layer or hierarchical latent spaces, attention-independent grounding objectives, and stronger validation against retraining-based influence.

Latent features may remain partially entangled, and attribution can depend on the chosen layer and dictionary capacity; phrase-region alignment relies on pretrained attention as a correspondence signal, which may be noisy or incomplete; EK-FAC and latent-mediated influence remain approximations to the effect of full retraining. The authors also report that moving the sparse autoencoder from layer 19 to layer 12 drops GQA accuracy substantially, showing that layer placement materially affects representation fidelity.