Causal pairs pulled from nuclear event reports now carry a confidence score: reviewing the top 20% catches 82% of extraction errors
Synopsis
Large language models extract cause-effect pairs from nuclear Licensee Event Reports as flat assertions, leaving risk analysts unable to separate reliable relations from spurious ones. A study stacks four signals, token-level probabilities, self-consistency across stochastic decoding runs, disagreement among three independently fine-tuned models, and conformal prediction, into a calibrated confidence score per pair. On a held-out test set the integrated score reached an expected calibration error of 0.067 and an AUROC of 0.84, above every individual signal, and uncertainty-guided review found 82% of extraction errors within the first 20% of relations reviewed, against 24% for random selection.
Interpretation
Causal extractions from nuclear event reports can now carry a comparable confidence value rather than only the cause-effect pair itself. Earlier extraction systems emitted every causal relation as a deterministic assertion, so a reliable extraction and a fabricated one were indistinguishable in representational form. On the held-out test set the integrated score reached an expected calibration error of 0.067 and an AUROC of 0.84, against 0.092 and 0.71 for token-level probability, 0.78 for self-consistency, 0.76 for ensemble disagreement and 0.73 for the conformal signal.
The confidence value is used to decide which extractions go to expert review first and which relations may enter a probabilistic risk model. Without uncertainty information, review could only be random or rest on heuristics that are neither systematic nor auditable. On 300 relations annotated by two experts each, the integrated score correlated -0.52 with expert validity judgments and reached an AUROC of 0.81 for predicting them; expert-judged validity was 0.94, 0.82 and 0.61 in the low, medium and high uncertainty bins.
The method combines four complementary uncertainty signals into one probability instead of relying on a single indicator. Token-level probability was better calibrated but weaker at discrimination, while self-consistency and ensemble disagreement discriminated better but calibrated worse, each covering only part of the information. A logistic regression stacker trained on the calibration set maps the four signals to a probability of correct extraction, and its risk-coverage area is 0.89 against 0.82 for the best individual signal.
Selecting relations by uncertainty before updating a risk model lands closer to the expert-validated reference than incorporating every extraction. Incorporating all 1,247 relations left the core damage frequency estimate 14.2% away from the expert-validated reference. Selecting the 728 relations with integrated score below 0.3 gave a 4.8% deviation, random selection of the same 728 gave 9.7%, and the expert-validated reference itself is 0.0%.
Perspective
The evaluation is confined to a corpus of U.S. commercial nuclear power plant Licensee Event Reports, with extraction models following the prior fine-tuned large language model route on that corpus and expert annotation covering 300 relations judged by eight experts. The framework is described as adaptable to other safety-critical domains, but the empirical work is nuclear only.
The authors note that other domains, languages and document types may require recalibrating the signals and their integration; the 300-relation annotation sample leaves the precision of the correlations and generalization to the wider extraction population subject to sampling uncertainty; and the decision evaluation uses a simplified risk model that omits dimensions such as importance measures and identification of risk-significant pathways. The confidence measures whether an extraction faithfully reflects what the source text states, not whether the stated relation is true in the world.