Patient-safety reasoning leans on a knowledge graph: removing it drops F1 by 0.047 and worsens all three metrics
Synopsis
A patient-safety reasoning system combines a knowledge graph, retrieval-augmented generation and a LLaMA model to extract safety concepts from MIMIC-III and MIMIC-IV records and produce traceable explanations. The complete system reached an F1 of 0.89 for concept extraction; removing the knowledge graph lowered F1 by 0.047, AUROC by 0.084 and AUPRC by 0.110, all significant. Removing retrieval or LoRA adaptation gave smaller, non-significant drops, and retrieval relevance was confirmed only qualitatively by experts.
Interpretation
Automated recognition of patient-safety concepts can now rest on a structured safety knowledge graph rather than only on statistical associations a language model learns from note text. Earlier patient-safety NLP mostly classified or extracted concepts, knowledge graphs emphasized representation and relational reasoning, and retrieval-augmented systems served question answering or general decision support; the three were rarely evaluated in one pipeline. On roughly 30,000 adult ICU stays, the complete system reached an F1 of 0.89 for patient-safety concept extraction; removing the knowledge graph lowered F1 by 0.047 (95% CI 0.015 to 0.079, bootstrap P=.004), AUROC by 0.084 and AUPRC by 0.110, all statistically significant.
The system first normalizes record information into standardized safety concepts, then passes graph relationships and retrieved literature to the language model to generate reasoning with sources attached. Normalization uses the ICPS patient-safety framework and maps differing clinical expressions onto shared concepts through SNOMED CT, UMLS and RxNorm where applicable. The retrieval corpus holds PubMed abstracts, patient-safety reports and clinical guidelines, encoded with BioBERT- or ClinicalBERT-based sentence embeddings and searched by similarity in a FAISS index; generation uses a LoRA-adapted LLaMA-based model, evaluated across MIMIC-III and MIMIC-IV.
Retrieval augmentation shows a consistent but non-significant gain in this configuration, acting more as an external evidence source than as a boost to extraction accuracy. Removing retrieval lowered F1 by 0.029, AUROC by 0.034 and AUPRC by 0.031, none statistically significant; removing LoRA changed F1 by only 0.006, also non-significant. Domain experts reviewed retrieved documents and confirmed their relevance to the evaluated patient-safety questions, but no quantitative retrieval precision, recall, citation accuracy or claim-level faithfulness was reported.
Perspective
The demonstrated scope is patient-safety concept extraction and evidence-supported reasoning on adult ICU stays from MIMIC-III and MIMIC-IV, with the system positioned as an advisory tool that clinicians review and whose final judgment remains with the clinical team. Usability was assessed on a simulated dashboard, where clinicians navigated alerts and explanations without difficulty and feedback recommended simplifying the interface and prioritizing high-level scores. The source proposes next steps of multi-center external validation, dedicated assessment of retrieval quality and claim-level faithfulness, and prospective clinical studies.
The concept-extraction evaluation set is relatively small and the cohort is adult ICU data, limiting generalization to pediatric, outpatient, community and other hospital settings. Retrieval augmentation was confirmed only qualitatively by expert relevance review, without quantified retrieval precision, recall, source quality, citation correctness or claim-level entailment. Usability was assessed in a simulated rather than real clinical workflow, and no prospective study has shown effects on clinical decision-making, patient-safety outcomes or clinician workload. System complexity and computational requirements may also pose implementation challenges for resource-constrained institutions.