# Vision-Language Models

Vision-language models lose accuracy when irrelevant images sit beside the target, and the loss persists even when the prompt names the target image, because middle-layer distractor-to-text connections carry the interference into reasoning. Using the single-image prediction as a reference, answer consistency regularization aligns answer distributions with and without distractors on Qwen2.5-VL, raising circular accuracy by an average of 16 percentage points across five perception benchmarks with five distractors while preserving single-image performance.

Vision-language models lose accuracy when irrelevant images sit beside the target, and the loss persists even when the prompt names the target image, because middle-layer distractor-to-text connections carry the interference into reasoning. Using the single-image prediction as a reference, answer consistency regularization aligns answer distributions with and without distractors on Qwen2.5-VL, raising circular accuracy by an average of 16 percentage points across five perception benchmarks with five distractors while preserving single-image performance.

Reasoning vision-language models must hold both multimodal context and a growing reasoning trace in their KV cache, and existing low-rank compression calibrates on static inputs, so its divergence from the dense model grows with generation length. TraceKV instead calibrates on reasoning traces the dense model generates itself, weights each KV state by its sensitivity to policy divergence before the SVD, and allocates ranks globally. Across four models including Qwen2.5-VL-7B, a 70% rank ratio with 4-bit quantization gives 73.41% overall accuracy versus 73.58% for the dense model, with the KV cache reduced 5.64 times.

Reasoning vision-language models must hold both multimodal context and a growing reasoning trace in their KV cache, and existing low-rank compression calibrates on static inputs, so its divergence from the dense model grows with generation length. TraceKV instead calibrates on reasoning traces the dense model generates itself, weights each KV state by its sensitivity to policy divergence before the SVD, and allocates ranks globally. Across four models including Qwen2.5-VL-7B, a 70% rank ratio with 4-bit quantization gives 73.41% overall accuracy versus 73.58% for the dense model, with the KV cache reduced 5.64 times.

ZoomGain is a data-construction pipeline that compares a full-image description of a region with a zoomed-in description of the same region and rewrites details visible only after zooming into questions about the whole image, verifying answers on the zoomed view with several models and discarding items answerable from language priors alone. On the resulting 1,000-question bilingual ZoomGainBench, the best of 28 vision-language models, GPT-5.6-Sol, scores only 61.08; after GRPO reinforcement learning on just 9K ZoomGain samples, Qwen3.5-9B averages 66.10 across 13 held-out benchmarks, above Qwen3.5-397B-A17B at 65.55.

ZoomGain is a data-construction pipeline that compares a full-image description of a region with a zoomed-in description of the same region and rewrites details visible only after zooming into questions about the whole image, verifying answers on the zoomed view with several models and discarding items answerable from language priors alone. On the resulting 1,000-question bilingual ZoomGainBench, the best of 28 vision-language models, GPT-5.6-Sol, scores only 61.08; after GRPO reinforcement learning on just 9K ZoomGain samples, Qwen3.5-9B averages 66.10 across 13 held-out benchmarks, above Qwen3.5-397B-A17B at 65.55.

Large-vision-language-model logical anomaly detection splits each constraint into a textual checklist, yet every check still reads the same raw image, so the model must pull different kinds of visual evidence from one input. CASCADE extends the decomposition to the image side, overlaying per-instance color for binding constraints or an object-anchored grid for calibration constraints, and CASCADE++ derives each scaffold's configuration from the constraint structure and the reference distribution. On MVTec-LOCO, CASCADE++ with three references and no training exceeds the strongest full-shot method and transfers across several vision-language backbones.

Large-vision-language-model logical anomaly detection splits each constraint into a textual checklist, yet every check still reads the same raw image, so the model must pull different kinds of visual evidence from one input. CASCADE extends the decomposition to the image side, overlaying per-instance color for binding constraints or an object-anchored grid for calibration constraints, and CASCADE++ derives each scaffold's configuration from the constraint structure and the reference distribution. On MVTec-LOCO, CASCADE++ with three references and no training exceeds the strongest full-shot method and transfers across several vision-language backbones.

Multimodal models answer questions and retrieve images, yet which image region and which word jointly push a given output score has been hard to quantify. MultiSHAP applies the Shapley Interaction Index from cooperative game theory to estimate, for every image-patch and text-token pair, a second-order synergy or suppression score, using only a query interface that allows separate perturbation of image and text and returns a task-relevant scalar. On 500 VQAv2 samples, deleting patches in the resulting ranking lowered the target-answer probability by 0.1720 on average, versus 0.0032 for the lowest-ranked patches.

Multimodal models answer questions and retrieve images, yet which image region and which word jointly push a given output score has been hard to quantify. MultiSHAP applies the Shapley Interaction Index from cooperative game theory to estimate, for every image-patch and text-token pair, a second-order synergy or suppression score, using only a query interface that allows separate perturbation of image and text and returns a task-relevant scalar. On 500 VQAv2 samples, deleting patches in the resulting ranking lowered the target-answer probability by 0.1720 on average, versus 0.0032 for the lowest-ranked patches.

Hierarchical multi-label classification of seafloor imagery has relied on low-level visual features such as color, texture and shape, leaving rare classes and shifting illumination hard to resolve. GHiML extracts concept variables tied to the training data from a vision language model, generates per-sample captions grounded in them, and fuses the resulting text embeddings with image embeddings through cross-attention and a per-node gated router. Across BenthicNet, FathomNet and iNaturalist, the router improves macro F1 over the vision-only baseline by up to 6.7 percentage points, and by up to 21.1 points on in-domain benthic data, with gains persisting under distribution shift.

Hierarchical multi-label classification of seafloor imagery has relied on low-level visual features such as color, texture and shape, leaving rare classes and shifting illumination hard to resolve. GHiML extracts concept variables tied to the training data from a vision language model, generates per-sample captions grounded in them, and fuses the resulting text embeddings with image embeddings through cross-attention and a per-node gated router. Across BenthicNet, FathomNet and iNaturalist, the router improves macro F1 over the vision-only baseline by up to 6.7 percentage points, and by up to 21.1 points on in-domain benthic data, with gains persisting under distribution shift.

Remote-sensing image fusion and semantic interpretation have usually been optimized separately, so good fusion metrics need not preserve downstream retrieval or detection utility. Lance maps each input to a closed-vocabulary text record of scene, objects and layout, conditions fusion on that language, re-describes the fused image as text, and back-propagates through text consistency and an anti-collapse text–mask contrastive loss with negative queries. Across seven test sets on GF2, QuickBird, WorldView-III and WorldView-II, reduced-scale reference metrics improve by up to 11.7% (median 2.1%) over the strongest learning baseline, and a held-out RemoteCLIP verifier plus a supervised DOTA detector give +6.4 mAP@0.5 over the closest competitor.

Remote-sensing image fusion and semantic interpretation have usually been optimized separately, so good fusion metrics need not preserve downstream retrieval or detection utility. Lance maps each input to a closed-vocabulary text record of scene, objects and layout, conditions fusion on that language, re-describes the fused image as text, and back-propagates through text consistency and an anti-collapse text–mask contrastive loss with negative queries. Across seven test sets on GF2, QuickBird, WorldView-III and WorldView-II, reduced-scale reference metrics improve by up to 11.7% (median 2.1%) over the strongest learning baseline, and a held-out RemoteCLIP verifier plus a supervised DOTA detector give +6.4 mAP@0.5 over the closest competitor.

Chest X-ray vision-language models can write fluent reports that nonetheless assert findings absent from the image, and earlier fixes mostly retrained the generator through reinforcement learning, preference optimization or chain-of-thought supervision. CheXPoT instead works after generation: it decomposes a report into checkable predicates, tests each against a visual evidence table built from a chest X-ray classifier and Grad-CAM locations, and rewrites the report after removing predicates the image cannot support. On a 4,145-image held-out split, MedGemma-4B reports moved from 19.2% to 46.8% micro-average precision and from 23.0% to 49.4% micro-average F1, with significant precision gains on 8 of 14 CheXpert classes and no significant decreases in precision or F1.

Chest X-ray vision-language models can write fluent reports that nonetheless assert findings absent from the image, and earlier fixes mostly retrained the generator through reinforcement learning, preference optimization or chain-of-thought supervision. CheXPoT instead works after generation: it decomposes a report into checkable predicates, tests each against a visual evidence table built from a chest X-ray classifier and Grad-CAM locations, and rewrites the report after removing predicates the image cannot support. On a 4,145-image held-out split, MedGemma-4B reports moved from 19.2% to 46.8% micro-average precision and from 23.0% to 49.4% micro-average F1, with significant precision gains on 8 of 14 CheXpert classes and no significant decreases in precision or F1.

Boundaries of subjective categories such as "healthy food" shift with culture and person, yet conventional workflows demand stable labels from the start. Google Research's Agile Deliberation splits classifier building into concept scoping and concept iteration: users break a broad concept into sub-categories and explicit exclusions, then a vision-language model surfaces semantically borderline images for the user to judge, with feedback routed into an automatic prompt-optimization loop. Across 18 live 1.5-hour sessions, classifiers built this way substantially outperformed the compared standard baselines on the most challenging subjective images, and participants reported less fatigue and anxiety.

Boundaries of subjective categories such as "healthy food" shift with culture and person, yet conventional workflows demand stable labels from the start. Google Research's Agile Deliberation splits classifier building into concept scoping and concept iteration: users break a broad concept into sub-categories and explicit exclusions, then a vision-language model surfaces semantically borderline images for the user to judge, with feedback routed into an automatic prompt-optimization loop. Across 18 live 1.5-hour sessions, classifiers built this way substantially outperformed the compared standard baselines on the most challenging subjective images, and participants reported less fatigue and anxiety.