Across five vision-centric tasks, adding labeled demonstrations to the context sharply raises accuracy on counterfactual examples that conflict with training priors while leaving prior-aligned accuracy nearly unchanged, indicating the models already have the visual capability and were being overridden by learned regularities.
Using a controlled two-stage PGD attack on 200 held-out COCO images with Qwen2.5-VL-7B-Instruct, the authors show that a targeted perturbation can drive teacher-forced training loss for a fixed target caption to near zero while free generation still produces the original correct description, localize this train/inference gap to a single autoregressive step (target-token rank fixed at exactly 3,488 of 152,064 vocabulary entries with zero variance), and trace all 28 decoder layers to show the visual encoder corrupts every image by a comparable margin while the language decoder amplifies the corrupted signal for susceptible images and actively suppresses it for resistant ones (p<0.001, rank-biserial r=0.579).
Across four vision-language models (Qwen3-VL-2B, Qwen3-VL-8B, Molmo2-7B, Llava-Next-34B), the work identifies attention heads causally necessary for OCR and shows they are general-purpose verbalization heads; summing their output-value matrices yields a verbalization lens that decodes image-token hidden states into interpretable semantic words at any layer, including layer 0, and whose pseudo-inverse provides concept vectors for editing image representations (e.g., replacing a tractor with a revolver), supporting the view that image representations are aligned with language space from early layers.
You’re all caught up