Vision-language models stop pruning whole layers: head-level attribution cuts CHAIR-i from 18.0 to 8.6 on multi-label COCO
Synopsis
Vision-language models often describe objects that are not in the image, and earlier fixes intervened at the output level or across whole attention layers, which cannot separate heads that support faithful captions from heads that drive hallucination. HACP scores each attention head at inference time by the spectral change of its perturbed representation, then suppresses hallucination-prone heads and reinforces faithful ones during decoding without training. On a 1000-image multi-label COCO subset it lowers CHAIR-s/i from 66.7/18.0 to 43.4/8.6 and raises F1 from 73.0 to 77.0.
Interpretation
The point of intervention moves from the output layer and whole attention layers down to individual attention heads: heads judged to drive hallucination are suppressed while heads judged to support grounded description are reinforced within the same layer. Earlier mitigation relied on output-level or coarse-grained interventions that could not selectively regulate heads tied to faithful versus hallucinatory generation, often at the cost of descriptive coverage. On 1000 randomly selected multi-label COCO images, HACP lowers CHAIR-s/i from 66.7/18.0 under Vanilla to 43.4/8.6 and raises F1 from 73.0 to 77.0; recall 70.8 stays close to Vanilla's 69.1, while precision rises from 77.3 to 89.0.
Each head's faithful or hallucinatory tendency is scored from the spectral change of its perturbed semantic representation rather than from a shift in the output distribution alone. Output-level measures such as KL divergence capture only distributional shift and cannot say whether the change improves semantic alignment, since faithful and hallucination-prone heads can both score high. InfoSpectralScore combines a dominant-variance EigenScore with a spectral-variance term that penalizes anisotropy and a spectral-entropy term that measures how concentrated the spectrum is, computed after mean-replacement perturbation of each head in the target layers and thresholded by quantile into faithful and hallucinatory sets.
After scoring, heads are not simply zeroed; their outputs are added or subtracted along a direction during decoding, with the strength searched per input. Conventional pruning such as zero or mean ablation is often too severe, over-penalizing the model and degrading semantic expressivity. The mechanism is implemented with inference-time forward hooks that add a smoothed, ratio-scaled set-mean direction to faithful heads and subtract it from hallucinatory heads without extra parameters; the per-image pipeline runs a small Bayesian optimization over the modulation strength and smoothing factor for the current image-prompt pair.
Perspective
Evaluation centers on instance-level object hallucination, meaning mentions of objects absent from the image; the models tested are LLaVA-1.5-7B/13B, Shikra-7B and Qwen2.5-VL-7B-Instruct, on CHAIR over COCO, the full POPE benchmark and MME. Attribute and spatial hallucinations fall outside the current evaluation.
The authors note that architectural differences may affect which heads are selected and how well the intervention transfers beyond the evaluated models; the full task-specific pipeline adds per-instance attribution and Bayesian search, requiring repeated decoding, while the fixed-configuration variant shares one pruning configuration across inputs.