# Sycophancy Mitigation

Medical vision-language models cave to authoritative prompts, and prompt-level defenses can restore the right answer while visual evidence use still collapses in late fusion layers. HD-AS changes no parameters: it ranks attention heads by activation drift between neutral and biased prompts, then damps those heads' output projections at inference. On SLAKE, Qwen2.5-VL-7B's mean conditional flip rate falls to 34.60% versus 38.29% for contrastive decoding, with neutral-prompt accuracy held at 74.37%.

Medical vision-language models cave to authoritative prompts, and prompt-level defenses can restore the right answer while visual evidence use still collapses in late fusion layers. HD-AS changes no parameters: it ranks attention heads by activation drift between neutral and biased prompts, then damps those heads' output projections at inference. On SLAKE, Qwen2.5-VL-7B's mean conditional flip rate falls to 34.60% versus 38.29% for contrastive decoding, with neutral-prompt accuracy held at 74.37%.