# Attention Steering

The internal choices inside a visual-head discovery score reshape which heads it returns in vision-language models. On Qwen3-VL-8B-Instruct, fully crossing query type, discovery corpus and the token positions whose attention is read, and testing all 12 head-sets with steerability and necessity interventions, only necessity follows the attention source: query-span head-sets are more necessary than last-token ones, and the two referring plus query-span sets lie 17–19 standard deviations above 250 layer-matched random sets. At the default strength, no discovered head-set steers better than its layer-matched random sets.

The internal choices inside a visual-head discovery score reshape which heads it returns in vision-language models. On Qwen3-VL-8B-Instruct, fully crossing query type, discovery corpus and the token positions whose attention is read, and testing all 12 head-sets with steerability and necessity interventions, only necessity follows the attention source: query-span head-sets are more necessary than last-token ones, and the two referring plus query-span sets lie 17–19 standard deviations above 250 layer-matched random sets. At the default strength, no discovered head-set steers better than its layer-matched random sets.

Revisiting visual attention sinks in vision-language model decoders, this work finds a layer-dependent structure in which early and final decoder layers repeatedly collapse onto the same few image regions across prompts (Prompt-Invariant Sinks, PIS) while mid layers stay prompt-conditioned and drive vision-language alignment; it then proposes SAGE, a training-free intervention that applies an additive bias to visual-token attention logits using token-aligned ROI masks from standard vision backbones such as CLIP, ViT, and DINOv3, steering attention away from PIS toward query-relevant regions and improving visual grounding, reducing hallucination, and yielding gains on public benchmarks across LLaVA-1.5, Qwen2-VL, InternVL3, and NVILA.

Revisiting visual attention sinks in vision-language model decoders, this work finds a layer-dependent structure in which early and final decoder layers repeatedly collapse onto the same few image regions across prompts (Prompt-Invariant Sinks, PIS) while mid layers stay prompt-conditioned and drive vision-language alignment; it then proposes SAGE, a training-free intervention that applies an additive bias to visual-token attention logits using token-aligned ROI masks from standard vision backbones such as CLIP, ViT, and DINOv3, steering attention away from PIS toward query-relevant regions and improving visual grounding, reducing hallucination, and yielding gains on public benchmarks across LLaVA-1.5, Qwen2-VL, InternVL3, and NVILA.