Medical vision-language models get a calibration stress test: 1,638 controlled runs expose where confidence breaks
Synopsis
Evaluation of medical vision-language models has centered on accuracy, leaving whether their confidence is trustworthy largely untested. MVC-Bench splits calibration into three axes, robustness, effectiveness of calibration and prompt-tuning methods, and stability under prompt templates and random seeds, spanning fundus, histopathology and chest X-ray modalities, eight backbones and six prompt-tuning methods across 1,638 controlled experiments. In-domain calibration rankings barely survive domain shift, with a mean Kendall correlation of -0.0598 over 30 rank-transfer tests and none significantly positive; the proposed Multi-Class Margin regularizer reaches the lowest ECE in 10 of 12 in-domain settings but only 4 under domain shift.
Interpretation
Confidence calibration for medical vision-language models now has a dedicated evaluation protocol rather than an accuracy-only leaderboard. Prompt-tuning methods have mainly optimized accuracy, leaving calibration underexplored in the medical domain, while clinical deployment requires confidence that reflects the true likelihood of correctness. The benchmark covers eight backbones across three modalities in in-domain and domain-shift settings, totaling 1,638 seed-level controlled runs, with accuracy and 20-bin ECE as primary metrics and MCE and ACE as complementary measures.
Calibration-method rankings chosen in-domain largely fail to carry over to external datasets. Whether in-domain calibration rankings predict domain-shift behavior had not been directly tested. Across 30 in-domain-to-domain-shift rank-transfer tests, Kendall tau-b has a mean of -0.0598 and a median of -0.0476, with 17 negative and 13 positive correlations and none significantly positive at p < 0.05.
Calibration error and accuracy have no universal alignment; the direction depends on both calibration method and backbone. Accuracy alone cannot show whether a model stays confidently wrong under class imbalance or prevalence shift. Pearson correlations between accuracy and ECE per method and backbone are often negative for TS, MDCA and MBLS, consistently positive for ECCV_Penalty, and reach extreme plus or minus one for X-ray backbones because of limited dataset pairs.
A multi-class margin regularizer pushes samples from low-margin regions toward moderate margins and lowers calibration error in most in-domain settings. Reliability diagrams show underconfidence arising where the correct-class logit only slightly exceeds competing logits, and MBLS reduces extreme margins without lifting small ones. Across 12 modality-specific settings, MCM achieves the lowest ECE in 10 in-domain settings and in 4 under domain shift; paired Wilcoxon tests give p = 0.0212 against ECCV_ZS and p = 0.0005 against ECCV_Penalty, with no significant advantage under domain shift.
Perspective
The evaluation is limited to medical image classification, covering in-domain and domain-shift settings for fundus, histopathology and chest X-ray modalities; segmentation, detection, report generation, retrieval and medical visual question answering use different uncertainty definitions and evaluation protocols and are not included. MCM hyperparameters are selected on validation data only and then fixed across the remaining backbones, modalities and domain-shift evaluations, and a four-fold leave-one-dataset-out analysis over the fundus datasets independently selects the same setting in every fold.
MCM targets low-margin underconfidence, and in PLIP-Histo. and QuiltNet-Histo. overconfidence is more prominent in that modality, so ECE increases; under domain shift MCM is not uniformly superior and no paired comparison reaches significance. The authors also note that modality-specific margin behavior and calibration objectives under broader clinical shifts need further study.