Visual-head discovery scores shift with where attention is read: necessity tracks the query span, while steering at default strength is matched by random heads
Synopsis
The internal choices inside a visual-head discovery score reshape which heads it returns in vision-language models. On Qwen3-VL-8B-Instruct, fully crossing query type, discovery corpus and the token positions whose attention is read, and testing all 12 head-sets with steerability and necessity interventions, only necessity follows the attention source: query-span head-sets are more necessary than last-token ones, and the two referring plus query-span sets lie 17–19 standard deviations above 250 layer-matched random sets. At the default strength, no discovered head-set steers better than its layer-matched random sets.
Interpretation
Within one attention-scoring pipeline, changing which token positions' attention is read, or how the question is phrased, swaps out most of the selected visual heads, while changing the corpus matters least. Visual heads, gaze heads and visual retrieval heads had each been reported under one fixed setting, so their claims could not be compared, and the choices inside the discovery score had not been varied systematically. On Qwen3-VL-8B-Instruct, two random halves of the same discovery data share 96–99 of the top-100 heads; changing only the attention source keeps a median of 49 of the top-100 heads, the query type 55 and the corpus 84.
Whether the model actually needs a head-set depends on where attention is read from, not on how the question is worded. The two published validations each compared against a single random draw, which cannot place a discovered set within the random distribution. All four query-span head-sets have necessity intervals above zero on both corpora and beat all 20 layer-matched random sets, while no last-token set does; the two referring plus query-span sets lie 17–19 control standard deviations above 250 random sets and survive Holm correction, and the effect replicates on Qwen3-VL-4B and Qwen2-VL-7B.
At the default intervention strength, steerability does not separate discovered head-sets from random ones. Prior work used a single random draw as the baseline, and a single draw is one sample from that distribution. No discovered head-set steers better than its layer-matched random sets, random sets from the same or from any layer span the discovered sets' scores, and single random draws range widely under layer matching.
The necessity effect comes from the question and option attention rows, not from the image rows or the final prompt token. Masking validations typically suppress a head-set's image attention as a whole, without separating which attention rows carry the effect. Masking only the question and option rows gives a necessity score as large as masking all rows for the referring plus query-span set, while masking only the image rows or the last row leaves the score near zero.
Perspective
The conclusions rest on the 12-setting design on Qwen3-VL-8B-Instruct, with second-model checks on Qwen3-VL-4B and Qwen2-VL-7B, out-of-domain results on five further models and pointing-suite results on eight. Steerability is evaluated at the default strength used in prior work, and masking is attention knockout. A reader applying these controls to another model would need to rebuild the layer-matched, many-draw baselines in the same way.
Steerability is evaluated only at the default strength; a pilot at weaker strength reported by the authors suggests several discovered sets may separate from random ones there, but that pilot uses only 5 random controls per setting and does not by itself establish specificity. Masking is attention knockout, with mean or resample ablation and activation patching as natural extensions. Out-of-domain and cross-model comparisons use one random draw per head-set. Image-attention mass is measured on a generic image prompt, and random head-sets span a narrower range of it than discovered sets.