# Answer Consistency Regularization

Vision-language models lose accuracy when irrelevant images sit beside the target, and the loss persists even when the prompt names the target image, because middle-layer distractor-to-text connections carry the interference into reasoning. Using the single-image prediction as a reference, answer consistency regularization aligns answer distributions with and without distractors on Qwen2.5-VL, raising circular accuracy by an average of 16 percentage points across five perception benchmarks with five distractors while preserving single-image performance.

Vision-language models lose accuracy when irrelevant images sit beside the target, and the loss persists even when the prompt names the target image, because middle-layer distractor-to-text connections carry the interference into reasoning. Using the single-image prediction as a reference, answer consistency regularization aligns answer distributions with and without distractors on Qwen2.5-VL, raising circular accuracy by an average of 16 percentage points across five perception benchmarks with five distractors while preserving single-image performance.