Jim Dorsey

Large language models extract cause-effect pairs from nuclear Licensee Event Reports as flat assertions, leaving risk analysts unable to separate reliable relations from spurious ones. A study stacks four signals, token-level probabilities, self-consistency across stochastic decoding runs, disagreement among three independently fine-tuned models, and conformal prediction, into a calibrated confidence score per pair. On a held-out test set the integrated score reached an expected calibration error of 0.067 and an AUROC of 0.84, above every individual signal, and uncertainty-guided review found 82% of extraction errors within the first 20% of relations reviewed, against 24% for random selection.

Large language models extract cause-effect pairs from nuclear Licensee Event Reports as flat assertions, leaving risk analysts unable to separate reliable relations from spurious ones. A study stacks four signals, token-level probabilities, self-consistency across stochastic decoding runs, disagreement among three independently fine-tuned models, and conformal prediction, into a calibrated confidence score per pair. On a held-out test set the integrated score reached an expected calibration error of 0.067 and an AUROC of 0.84, above every individual signal, and uncertainty-guided review found 82% of extraction errors within the first 20% of relations reviewed, against 24% for random selection.