Patient-safety reasoning without the knowledge graph loses 0.047 F1, while retrieval's gain straddles zero
Synopsis
Patient-safety reasoning requires linking clinical narratives, medications, laboratory findings and adverse events, and a language model alone can produce plausible but unsupported conclusions. The same MIMIC-III/IV cases ran under four conditions with one LLaMA 2 7B model and one prompt: model only, plus retrieval-augmented generation, plus a patient-safety knowledge graph and ontology, and both. The full framework reached F1 0.89 for concept extraction, 0.046 above baseline; removing the knowledge graph cost the most, with F1 0.047, AUROC 0.084 and AUPRC 0.110, while removing retrieval gave an F1 difference of 0.029 whose interval crossed zero.
Interpretation
Patient-safety reasoning is split into four comparable configurations that hold the model, cases, prompt and output requirements fixed while varying external evidence and structured knowledge. Earlier work typically evaluated a language model, a knowledge graph or retrieval augmentation on its own, or demonstrated one integrated system, rather than placing plain model reasoning beside each grounding mode under identical conditions. The comparison covers LLaMA 2 7B alone, with retrieval augmentation, with the knowledge graph and ontology, and with both, on cases drawn from MIMIC-III and MIMIC-IV, with a final cohort of approximately 30,000 adult ICU patients.
The full framework reached an F1 of 0.89 for patient-safety concept extraction and an AUROC of 0.87 for the broader predictive component on held-out data. Against the baseline, the full model improved F1 by an absolute 0.046 (95% CI 0.012 to 0.081, bootstrap p = 0.006), AUROC by 0.087 (0.045 to 0.127, DeLong p < 0.001) and AUPRC by 0.082 (0.017 to 0.144, bootstrap p = 0.011). In the same paired comparison, the McNemar test on the binary error pattern found no significant difference (p = 0.360), so the gains sit mainly in the continuous discrimination measures rather than in a single test.
Removing knowledge-graph grounding produced the largest drop among the ablations, making structured semantic relationships the most consequential grounding source in this framework. The full model versus the version without the knowledge graph showed an F1 difference of 0.047 (0.015 to 0.079, bootstrap p = 0.004), an AUROC difference of 0.084 (0.043 to 0.125, DeLong p < 0.001) and an AUPRC difference of 0.110 (0.044 to 0.175, bootstrap p = 0.001), with the paired McNemar test also significant (p = 0.029). Project documentation names knowledge-graph grounding as the component contributing most to concept extraction, at roughly 4 to 5 F1 percentage points, and the AUPRC gap is notable where positive safety concepts are less prevalent than negative cases.
Retrieval-augmented generation points in the positive direction, but its independent contribution is not precisely estimated in this ablation. Removing retrieval gave an F1 difference of 0.029 with a 95% CI of −0.003 to 0.059 (bootstrap p = 0.079), an AUROC difference of 0.034 (−0.006 to 0.073, DeLong p = 0.084) and an AUPRC difference of 0.031 (−0.036 to 0.098, bootstrap p = 0.375), with McNemar p = 0.190. Every reported difference favored the full model, and domain experts confirmed the retrieved documents were relevant to the evaluated clinical scenarios, but no formal measures of retrieval precision, recall, citation correctness or claim-level faithfulness are reported.
Perspective
The evaluation rests on de-identified retrospective critical-care data from MIMIC-III and MIMIC-IV, with inpatient and especially ICU settings, and its outcomes are model measures and expert ratings rather than patient outcomes.
Retrieval's gain has confidence intervals crossing zero on the principal continuous metrics, claim-level evidence attribution, retrieval recall and citation correctness are not formally measured, the knowledge graph and ontology depend on the completeness and currency of their sources, and usability testing ran only on a simulated dashboard without workflow or time-to-action assessment.