Deleting the waveform region an audio-language model's claim points to overturns its "sound present" answer more often than deleting an equally sized random window, indicating that these hallucinations are at least partly grounded in locatable audio evidence, and the same contrast lifts Qwen2-Audio accuracy from 57.1% to 71.9%.
The work introduces AnswerMap: the image is cut into K row and K column bands, each shown alone to a frozen VLM with a yes/no question about the query, and the outer product of the row and column "yes" posteriors gives a query-conditioned spatial map on which a fixed read-out (e.g., expectation, maximum) derives continuous outputs natively; across four models and three query distributions the map agrees with the model's own generated point at AUC 0.85 versus 0.38 for attention, deleting its region flips 53% of correct answers versus 19% for attention, and its maximum flags hallucinated objects, its expectation localizes when the model's own pointing fails, and its top-mass crop fixes about half of the model's wrong answers.
Using activation patching across twenty-five models spanning eight LLM families, the work identifies an early-layer (L0) attention routing circuit shared by VQ-tokenized vision-language models, proposes a three-gate diagnostic that isolates ten models carrying it and rejects the other fifteen, shows via a single-variable architectural swap (LLaVA-1.6 CLIP+MLP to VQ+Linear) that vector quantization is the source of the pathological signal, and finds that only L0 ablation reduces object hallucination in open-ended generation (CHAIR_i down 31% relative) while tuned DoLA and VCD do not.
You’re all caught up