# Medical Vision-Language Models

Evaluation of medical vision-language models has centered on accuracy, leaving whether their confidence is trustworthy largely untested. MVC-Bench splits calibration into three axes, robustness, effectiveness of calibration and prompt-tuning methods, and stability under prompt templates and random seeds, spanning fundus, histopathology and chest X-ray modalities, eight backbones and six prompt-tuning methods across 1,638 controlled experiments. In-domain calibration rankings barely survive domain shift, with a mean Kendall correlation of -0.0598 over 30 rank-transfer tests and none significantly positive; the proposed Multi-Class Margin regularizer reaches the lowest ECE in 10 of 12 in-domain settings but only 4 under domain shift.

Evaluation of medical vision-language models has centered on accuracy, leaving whether their confidence is trustworthy largely untested. MVC-Bench splits calibration into three axes, robustness, effectiveness of calibration and prompt-tuning methods, and stability under prompt templates and random seeds, spanning fundus, histopathology and chest X-ray modalities, eight backbones and six prompt-tuning methods across 1,638 controlled experiments. In-domain calibration rankings barely survive domain shift, with a mean Kendall correlation of -0.0598 over 30 rank-transfer tests and none significantly positive; the proposed Multi-Class Margin regularizer reaches the lowest ECE in 10 of 12 in-domain settings but only 4 under domain shift.

Medical vision-language models cave to authoritative prompts, and prompt-level defenses can restore the right answer while visual evidence use still collapses in late fusion layers. HD-AS changes no parameters: it ranks attention heads by activation drift between neutral and biased prompts, then damps those heads' output projections at inference. On SLAKE, Qwen2.5-VL-7B's mean conditional flip rate falls to 34.60% versus 38.29% for contrastive decoding, with neutral-prompt accuracy held at 74.37%.

Medical vision-language models cave to authoritative prompts, and prompt-level defenses can restore the right answer while visual evidence use still collapses in late fusion layers. HD-AS changes no parameters: it ranks attention heads by activation drift between neutral and biased prompts, then damps those heads' output projections at inference. On SLAKE, Qwen2.5-VL-7B's mean conditional flip rate falls to 34.60% versus 38.29% for contrastive decoding, with neutral-prompt accuracy held at 74.37%.

The work proposes ΔRepresentation, a visual phenotype representation learning framework for medical VLMs: BaseAnatomy applies geometry supervision to normal anatomical representations using ideal spatial distributions, while Phenotype uses counterfactual reasoning to estimate a normal anatomical reference for each pathological patch and learn phenotype-specific representation increments that cluster by phenotype. On ReXGroundingCT and LIDC-IDRI, it consistently improves lesion grounding and phenotype recognition across six pretrained medical VLMs.

The work proposes ΔRepresentation, a visual phenotype representation learning framework for medical VLMs: BaseAnatomy applies geometry supervision to normal anatomical representations using ideal spatial distributions, while Phenotype uses counterfactual reasoning to estimate a normal anatomical reference for each pathological patch and learn phenotype-specific representation increments that cluster by phenotype. On ReXGroundingCT and LIDC-IDRI, it consistently improves lesion grounding and phenotype recognition across six pretrained medical VLMs.