researchsquare.comSource publication:
Loading content…
Decisions in colorectal liver metastases weigh tumor biology, resectability and future liver remnant, and large language models previously produced recommendations from prompting alone with limited agreement with expert multidisciplinary practice. The METATRON study formalized the evaluation sequence hepatobiliary surgeons follow at tumor boards into a structured workflow and combined it with guideline retrieval, testing 110 synthetic vignettes scored by three independent LLM judges across five dimensions. The workflow-plus-retrieval strategy reached the highest composite score (0.626 versus 0.444 for baseline), yet sensitivity for anatomical resectability remained 40%, insufficient for autonomous clinical use.
The quality of LLM recommendations for colorectal liver metastases depends on the inference strategy, not only on the underlying model. Most prior work compared foundation models or prompting techniques rather than how clinical reasoning is organized, and prompting alone left the largest gaps in reasoning-related dimensions. Across 110 synthetic vignettes designed by hepatobiliary surgeons and covering eight clinical scenarios, each recommendation was scored against an expert reference by three independent judges, Claude Sonnet 4.6, GPT-4o and Llama 3.3 70B, and workflow-guided and retrieval-augmented strategies outperformed baseline under all three.
Writing the expert evaluation sequence explicitly into inference brings model recommendations closer to the multidisciplinary decision pathway. The baseline strategy asked the model to produce a recommendation directly, whereas the workflow strategy required sequential assessment of the primary tumor, hepatic disease, extrahepatic disease and anatomical feasibility before synthesis. The workflow was encoded as a tree-structured JSON schema with four reasoning domains A to D and conditional nodes activated only when clinically appropriate; across the 110 cases the workflow-plus-retrieval strategy achieved the highest normalized scores in completeness, step concordance, case tailoring and MDT decision concordance.
Retrieval augmentation and workflow guidance address different problems and are complementary rather than interchangeable. Retrieval mainly improved recognition of missing decision-critical information, while workflow guidance mainly improved reasoning coherence and agreement with expert recommendations. Retrieval scored highest on missing-data concordance (0.486, versus 0.419 for workflow-plus-retrieval and 0.130 for baseline), and macro F1 for anatomical resectability classification rose from 0.495 at baseline to 0.643 with workflow plus retrieval, though sensitivity for non-compliant cases was only 40%.
The evaluation used 110 synthetic vignettes designed by hepatobiliary surgeons and covering eight colorectal liver metastasis scenarios, and the system was assessed for methodological research only, not for real-time clinical decision-making. The authors propose prospective validation within real multidisciplinary tumor boards as the next step, evaluating consecutive cases against prospective MDT consensus and assessing effects on discussion efficiency and patient management.
Reference recommendations came from a single senior hepatobiliary expert and do not constitute an absolute gold standard; the three LLM judges differed in scoring calibration and no inter-rater agreement statistic was computed; sensitivity for anatomical resectability was 40%, which the authors state remains insufficient for autonomous clinical use; and all strategies used the same underlying model, gpt-oss-120B, leaving generalization across model backbones untested.