A council of small models lost to one general model: on 783 radiology reports, Qwen3-4B-2507 hit 91.69% and beat medical-tuned MedGemma
Synopsis
On 783 de-identified free-text mammography, hip-knee-ankle and brain MR reports, an LLM Council of three general small models plus a chairman reached 83.56% overall accuracy, below the medically trained MedGemma 4B at 88.72%, with the gap driven mainly by hip-knee-ankle classification. A single council member, Qwen3-4B-2507, reached the highest accuracy at 91.69%, above both the council and MedGemma, while the council needed four model calls per report and about 2.44 times MedGemma's peak GPU memory.
Interpretation
Consensus synthesis across several models did not beat a medically specialised model on radiology report extraction and classification, scoring 83.56% against 88.72% overall. Earlier comparisons mostly set single models against each other, leaving little direct evidence on council-style multi-model architectures versus a medical model on the same reports. On the same 783 reports, McNemar testing gave an overall p of 0.0129, with p = 0.0011 for hip-knee-ankle classification, while mammography and brain MR differences were not significant.
One general model inside the council, used alone, reached the highest accuracy, surpassing both the synthesised council output and MedGemma. The council combined LFM2.5-1.2B, Qwen3-4B-2507 and Phi-4-Mini-reasoning under chairman LLaMa 3.1 8B, and its synthesis did not preserve the strongest member's judgment. Qwen3-4B-2507 scored 91.89%, 92.93% and 90.25% on mammography, hip-knee-ankle and brain MR respectively, 91.69% overall with weighted F1 of 0.8902; its pooled 91.50% accuracy had a 95% confidence interval overlapping MedGemma's 89.51%.
The council's extra computation bought no accuracy gain, with four models in parallel costing about 4.05 times the parameters and four times the calls. MedGemma is a single 4B model with one call per report, whereas the council totals 16.2B parameters and four calls per report. Measured average inference time was 3.187 seconds per report versus 2.775 seconds, about 1.15 times, and average peak VRAM was 13.156 GB versus 5.391 GB, about 2.44 times.
Perspective
The findings rest on 783 de-identified reports from a single institution, Max Healthcare Saket, covering BI-RADS and ACR extraction, pre- versus post-operative hip-knee-ankle classification, and normal versus abnormal brain MR classification. The authors propose evaluating the council and individual models on larger multi-institutional datasets, testing different chairman models and alternative aggregation such as weighted voting, and extending to multimodal image-plus-report inputs and retrieval-augmented pipelines.
The authors note the results come from a single centre and text-only tasks, so they should not be read as general-purpose models beating medical models across all healthcare settings; the significant hip-knee-ankle gap drives the overall difference, while mammography and brain MR differences were not significant. Error analysis shows some reports lacked explicit textual cues about surgical status, and council synthesis sometimes failed to keep a correct member judgment.