# Semantic Entropy

The work introduces ProbeGuard, a certified abstention framework that decides from how multi-round consensus formed rather than from terminal agreement: it derives trajectory features from the system's execution log, checks unanimous first-round votes with rationale semantic entropy and an active counter-evidence probe, and converts these scores into a distribution-free selective-risk bound via stratified Learn-then-Test calibration; on MedQA terminal agreement scores 0.512 AUROC while the process score reaches 0.696, and 13.4% of unanimous votes are wrong, with the certified rule answering 55.3% of that layer at 9.0% observed risk.

The work introduces ProbeGuard, a certified abstention framework that decides from how multi-round consensus formed rather than from terminal agreement: it derives trajectory features from the system's execution log, checks unanimous first-round votes with rationale semantic entropy and an active counter-evidence probe, and converts these scores into a distribution-free selective-risk bound via stratified Learn-then-Test calibration; on MedQA terminal agreement scores 0.512 AUROC while the process score reaches 0.696, and 13.4% of unanimous votes are wrong, with the certified rule answering 55.3% of that layer at 9.0% observed risk.

The work tests semantic entropy as an escalation signal for routing queries from a small to a large language model across three benchmarks and two model families, finding that on GSM8K it reliably distinguishes the small model's mistakes (AUROC 0.871) and improves routed accuracy over random escalation by up to nine points at matched cost, while an equally strong-looking synthetic-arithmetic result is almost exactly matched by a model-free rule based only on how hard the question looks on the page; it then offers a checklist of controls, including a difficulty baseline, three escalation targets, an oracle-headroom pre-flight check, per-generation cost accounting, and a difficulty probe that predicted in advance a retrieval shortcut's collapse from AUROC 0.908 to 0.518.

The work tests semantic entropy as an escalation signal for routing queries from a small to a large language model across three benchmarks and two model families, finding that on GSM8K it reliably distinguishes the small model's mistakes (AUROC 0.871) and improves routed accuracy over random escalation by up to nine points at matched cost, while an equally strong-looking synthetic-arithmetic result is almost exactly matched by a model-free rule based only on how hard the question looks on the page; it then offers a checklist of controls, including a difficulty baseline, three escalation targets, an oracle-headroom pre-flight check, per-generation cost accounting, and a difficulty probe that predicted in advance a retrieval shortcut's collapse from AUROC 0.908 to 0.518.