A medical vision-language model that scores 0.895 on diagnosis and points back to the lesion

A medical vision-language model now returns a diagnosis, a free-text answer and a pixel-level localization of the finding, together with a stepwise textual explanation. Earlier medical foundation models were largely modality specific, multimodal models underperformed on pixel-precision tasks such as segmentation and medical visual grounding, and most remained black boxes without stated evidence. On internal diagnostic validation EVLF-FM reached the highest mean accuracy among all evaluated models, 0.895, beating every generalist vision-language model on both accuracy and F1; across 11 datasets it had the highest accuracy over specialist deep learning models on six and over specialist foundation models on four.

The model encodes whole-image disease features and spatially detailed pixel features separately, then projects both into the language model's representation space so reasoning and localization share one input. Conventional pipelines pool the image into a single global vector or emit only a final prediction, leaving no visual evidence tied to a region. Training drew on more than 1.3 million samples from 18 global datasets across 10 imaging modalities; external validation used 13,443 independent test samples from 17 additional datasets across 10 modalities.

Training runs in two stages: low-rank-adaptation supervised fine-tuning, then GRPO reinforcement learning whose reward combines diagnostic correctness, localization intersection over union and sentence-level clinical reasoning alignment. Reinforcement learning without supervised initialization degraded sharply to 0.676/0.531 accuracy/F1; supervised fine-tuning alone gave 0.845/0.769, and adding GRPO raised it to 0.858/0.797. Twelve physicians, four junior and eight specialists in radiology, pathology and ophthalmology, graded reasoning outputs with the S.C.O.R.E. framework, giving an overall mean of 4.25/5, with objectivity highest at 4.36.

The demonstrated scope is diagnosis, medical visual question answering and pixel-level grounding on static two-dimensional images, with training and test data drawn from public, controlled-access and access-restricted clinical datasets covering six specialties and 10 imaging modalities; the authors position it as a generalizable multimodal backbone for further downstream fine-tuning rather than a deployable system.

The authors state that current accuracy does not yet reach the level typically required for direct clinical deployment and that VQA performance is generally lower than diagnostic classification; the model handles only static two-dimensional images and does not explicitly process volumetric data; the training set, though large, captures a limited portion of real-world diversity, so predictions may be less reliable for underrepresented groups, devices or rare pathologies; they call for prospective clinical studies of safety, effectiveness and clinical utility before routine deployment.