Picking the right image is not enough: Qwen2.5-VL gains 16 circular-accuracy points with five distractors from answer-consistency training
Synopsis
Vision-language models lose accuracy when irrelevant images sit beside the target, and the loss persists even when the prompt names the target image, because middle-layer distractor-to-text connections carry the interference into reasoning. Using the single-image prediction as a reference, answer consistency regularization aligns answer distributions with and without distractors on Qwen2.5-VL, raising circular accuracy by an average of 16 percentage points across five perception benchmarks with five distractors while preserving single-image performance.
Interpretation
Accuracy lost to irrelevant images cannot be attributed entirely to picking the wrong image: naming the target image in the prompt still leaves the answering accuracy depressed. Earlier mitigations focused on selecting relevant images or limiting information exchange between images, implicitly assuming correct selection removes the interference. Across three open 7–8B vision-language models and five perception benchmarks augmented with distractors, naming the target with five distractors recovered none of the circular-accuracy drop; averaged over the three models and five benchmarks, mean-rotation accuracy fell 13.3 points under real distractors versus 3.2 points when they were replaced by same-size black images.
Distractor information reaches reasoning mainly through middle-layer connections from distractor images into text representations, not only through image-to-image links. Masks that restrict attention between images leave question tokens free to attend to distractor images, so a residual pathway stays open. Severing distractor-to-text connections in layers 9–17 of Qwen2.5-VL recovered 47% of distraction-broken examples while damaging 5% of correct ones; inter-image masking in the same layers recovered 10% while damaging 57% of correct outputs.
Answer consistency regularization anchored to the clean single-image prediction improves robustness without directly constraining attention. Hard pathway blocking needs to know which images are distractors at test time, whereas the training objective does not. With LoRA fine-tuning of Qwen2.5-VL-7B-Instruct, pooled circular accuracy with five distractors was 69.6%, against 59.3% for the base model and 66.6% for a cross-entropy-only control; under target-naming prompts the gain stays at 15.4–16.8 points at every distractor count.
Perspective
The experiments are confined to one target image plus randomly sampled distractors: 9,052 questions from five perception benchmarks, three open 7–8B models for behavioral analysis, and mechanistic knockouts and fine-tuning only on Qwen2.5-VL-7B-Instruct, with fine-tuning data from FineVision domains different from the evaluation benchmarks.
Evaluation uses randomly sampled distractors, leaving open how specific distractor properties affect robustness; the authors call for structured spatial distractors to separate perceptual confusion from spatial-reasoning failure, and for extending the framework to video to test dynamic distractor contexts.