CheXPoT checks chest X-ray reports after generation: MedGemma-4B micro-average precision rises from 19.2% to 46.8%

Chest X-ray report generation gains a post-hoc visual check: each finding in a report is decomposed into predicates that can be judged one by one, only predicates supported by image evidence are kept, and a language model rewrites the report after the rest are removed. Earlier approaches mostly worked on the generation side, retraining models with reinforcement learning, preference optimization or chain-of-thought supervision at substantial compute cost; existing hallucination detectors such as ReXTrust give fine-grained scores for suspect findings but do not directly produce a corrected report for downstream use. On a 4,145-image held-out test split, scored with CheXbert-extracted labels, MedGemma-4B reports moved from 19.2% to 46.8% micro-average precision and from 23.0% to 49.4% micro-average F1; paired nonparametric bootstrap 95% confidence intervals over 10,000 resamples showed significant precision gains on 8 of the 14 CheXpert classes and significant F1 gains on 5, with no statistically significant decreases in precision or F1.

The check runs in four steps: decompose the report into predicates, build a visual evidence table, answer each predicate with program-of-thought code, then rewrite the report from those answers. Decomposition uses RadGraph to extract clinical entities from the report and gpt-4o-mini to turn the entity list into predicates; the visual evidence table comes from a DenseNet-121 fine-tuned on the CheXpert Plus training split, which outputs probabilities for the 14 CheXpert labels, converts them to presence calls at per-class thresholds chosen for best validation F1, and uses Grad-CAM to extract a representative point for each class. For program-of-thought verification, gpt-4.1 generates query code from the predicates and the table structure but not its contents, and the table contents are passed through only when the code executes, with inconclusive answers discarded; gpt-4o-mini then compares the original report with the new answers and corrects the report or leaves it unchanged when the information matches. Running the pipeline over all 4,145 reports took 618 minutes and $26.77 in OpenAI API spend, averaging 8.95 seconds and $0.00646 per report.

The same correction pipeline also ran on two other report generators, and corrected reports kept the style and reporting format of their own generator. On LLaVA-Med, micro-average precision rose from 4.5% to 17.7% and micro-average F1 from 6.3% to 19.4%; on MAIRA-2, micro-average precision rose from 15.6% to 42.1% and micro-average F1 from 21.0% to 45.3%. All three generators were compared report by report on the same 4,145-image held-out test split under the same evaluation protocol; the variation in gains across models is read as support for correcting from the original report rather than overwriting it with the visual findings.

All experiments use the single CheXpert Plus dataset, and evaluation computes precision, recall and F1 from 14 CheXpert labels extracted by CheXbert rather than from direct radiologist review of the reports; correction is also limited by the DenseNet-121 verifier, so errors in the visual evidence table can propagate into the rewritten report.

The results are presented as evidence that post-hoc visual verification is feasible for MedGemma-4B outputs, not as proof of clinical effectiveness; the authors note that more precision gains than F1 gains is consistent with a precision-recall tradeoff, since stricter verification can remove hallucinated findings but may also remove some true positives. The pipeline relies on code generated by a language model, no code sandboxing was performed in the experiment, and no code errors or malicious-code incidents were observed; CheXPoT has not been tested in a clinical workflow, and a blind radiologist review would be needed to assess whether visually verified rewrites improve trust, efficiency or diagnostic safety.