A three-legged peacock still gets "two legs": a few in-context examples move attention off the category prior and onto the pixels

Vision-language models fail when an image violates regularities seen in training, such as answering "two" for a three-legged peacock; these errors can be corrected at inference time with in-context demonstrations rather than architectural changes or retraining. Most prior remedies treated such failures as missing visual capability and intervened mechanistically, changed the training recipe, or improved input preprocessing, while other work reported that visual in-context learning yields limited benefit. On a controlled benchmark that separates prior-consistent from prior-conflicting examples, Qwen3-VL 235B gains about 30 percentage points on counterfactual examples at 16 shots while prior-aligned accuracy stays effectively unchanged, and the 32B model shows the same asymmetry.

Demonstrations work by shifting the visual evidence that answer generation draws on, away from image tokens that support the learned prior and toward tokens that support the current image. The study assigns approximate semantic labels to image tokens with logit lens and LatentLens, marking tokens that decode to "peacock" as prior cues and tokens that decode to visible legs as grounding cues, with the two sets made disjoint by construction. Across Qwen3-VL 4B and 32B and Gemma-4 E4B and 31B, answer-token attention to prior cues falls and attention to grounding cues rises after demonstrations, and masking prior cues beats a matched random mask on all five tasks in both larger models.

In-context learning can reselect a reliable capability that a prior was suppressing, but it does not make an unreliable capability robust. If prior conflict were the only obstacle, demonstrations should help uniformly across leg counts; instead gains shrink for 4- and 5-leg cases. On 450 synthetic 2D-shape counting images with 50 examples for each count from 8 through 16, demonstration gains vary across counts and counts 10 and 15 fall below zero-shot performance.

The diagnostic targets researchers and evaluation designers studying visual grounding in VLMs, validated on five closed-label vision-centric tasks: counting, color identification, shape identification, object direction, and spatial relation, across Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5, and Gemma-4. The complex setting contains 1,160 examples, 650 real and 510 counterfactual, plus an expanded version with roughly 400 complex examples per task. The authors note that in-context learning is a training-free but not compute-free path to adaptation: the interleaved 32-shot setting consumes substantial context, increases memory use, and adds latency, so practical continual-adaptation systems will need to select demonstrations carefully, compress or cache visual contexts, or distill successful contexts into model weights.

The authors report that in-context learning is limited when the underlying visual capability is weak or absent; high-cardinality counting remains uneven even after animal identity and biological priors are removed, with counts 10 and 15 falling below zero-shot. Object direction and spatial relation have no single prior answer, so the mechanistic analysis uses the object named in the caption as a proxy cue, making that evidence less direct than for counting, color, and shape. The real and counterfactual subsets of object direction use disjoint labels, so the task is interpreted as a rare-orientation stress test rather than a fully label-matched comparison.