Jeonghyo Song

Revisiting visual attention sinks in vision-language model decoders, this work finds a layer-dependent structure in which early and final decoder layers repeatedly collapse onto the same few image regions across prompts (Prompt-Invariant Sinks, PIS) while mid layers stay prompt-conditioned and drive vision-language alignment; it then proposes SAGE, a training-free intervention that applies an additive bias to visual-token attention logits using token-aligned ROI masks from standard vision backbones such as CLIP, ViT, and DINOv3, steering attention away from PIS toward query-relevant regions and improving visual grounding, reducing hallucination, and yielding gains on public benchmarks across LLaVA-1.5, Qwen2-VL, InternVL3, and NVILA.

Revisiting visual attention sinks in vision-language model decoders, this work finds a layer-dependent structure in which early and final decoder layers repeatedly collapse onto the same few image regions across prompts (Prompt-Invariant Sinks, PIS) while mid layers stay prompt-conditioned and drive vision-language alignment; it then proposes SAGE, a training-free intervention that applies an additive bias to visual-token attention logits using token-aligned ROI masks from standard vision backbones such as CLIP, ViT, and DINOv3, steering attention away from PIS toward query-relevant regions and improving visual grounding, reducing hallucination, and yielding gains on public benchmarks across LLaVA-1.5, Qwen2-VL, InternVL3, and NVILA.