# VLM Interpretability

The work introduces AnswerMap: the image is cut into K row and K column bands, each shown alone to a frozen VLM with a yes/no question about the query, and the outer product of the row and column "yes" posteriors gives a query-conditioned spatial map on which a fixed read-out (e.g., expectation, maximum) derives continuous outputs natively; across four models and three query distributions the map agrees with the model's own generated point at AUC 0.85 versus 0.38 for attention, deleting its region flips 53% of correct answers versus 19% for attention, and its maximum flags hallucinated objects, its expectation localizes when the model's own pointing fails, and its top-mass crop fixes about half of the model's wrong answers.

The work introduces AnswerMap: the image is cut into K row and K column bands, each shown alone to a frozen VLM with a yes/no question about the query, and the outer product of the row and column "yes" posteriors gives a query-conditioned spatial map on which a fixed read-out (e.g., expectation, maximum) derives continuous outputs natively; across four models and three query distributions the map agrees with the model's own generated point at AUC 0.85 versus 0.38 for attention, deleting its region flips 53% of correct answers versus 19% for attention, and its maximum flags hallucinated objects, its expectation localizes when the model's own pointing fails, and its top-mass crop fixes about half of the model's wrong answers.