# Counteractive Pruning

Vision-language models often describe objects that are not in the image, and earlier fixes intervened at the output level or across whole attention layers, which cannot separate heads that support faithful captions from heads that drive hallucination. HACP scores each attention head at inference time by the spectral change of its perturbed representation, then suppresses hallucination-prone heads and reinforces faithful ones during decoding without training. On a 1000-image multi-label COCO subset it lowers CHAIR-s/i from 66.7/18.0 to 43.4/8.6 and raises F1 from 73.0 to 77.0.

Vision-language models often describe objects that are not in the image, and earlier fixes intervened at the output level or across whole attention layers, which cannot separate heads that support faithful captions from heads that drive hallucination. HACP scores each attention head at inference time by the spectral change of its perturbed representation, then suppresses hallucination-prone heads and reinforces faithful ones during decoding without training. On a 1000-image multi-label COCO subset it lowers CHAIR-s/i from 66.7/18.0 to 43.4/8.6 and raises F1 from 73.0 to 77.0.