# Probability Calibration

3 related articles

Subscribe to updates

arXiv Oct 12, 2026
Teaching a model to say "I don't know": a separate answer selector beats retraining the policy, lifting TriviaQA coverage from 20.2% to 30.9%
On factual question answering with Qwen3-14B, keeping the generator fixed and training a separate answer selector works better than training the model itself when to abstain. On TriviaQA the selector raises mean coverage from 20.2% under the original calibrated confidence filter to 30.9%, while pooled error among scored answers falls from 23.0% to 14.5%; on NQ-Open its mean utility still stays below always abstaining, so reliable cross-dataset abstention remains unresolved.
On factual question answering with Qwen3-14B, keeping the generator fixed and training a separate answer selector works better than training the model itself when to abstain. On TriviaQA the selector raises mean coverage from 20.2% under the original calibrated confidence filter to 30.9%, while pooled error among scored answers falls from 23.0% to 14.5%; on NQ-Open its mean utility still stays below always abstaining, so reliable cross-dataset abstention remains unresolved.
Autonomous University of Queretaro Instituto Politécnico Nacional
Juan R. Terven（First author）
Abstention Selective Prediction Qwen3 Probability Calibration Reinforcement Learning
arXiv Sep 30, 2026
Porcedda's Sys1Cal-v1 shows Jev's Choice probabilities are systematically distorted, and restoring an uncertainty component lifts median soft accuracy from 0.771 to 0.978
Porcedda built Sys1Cal-v1, a synthetic dataset of 365 True/False items derived from 92 probability problems whose proposition probabilities are known by construction, and used total variation distance and distributional overlap (soft accuracy) to evaluate Jev's Noul, Choice, and Score primitives plus the open-source baseline SemIf, finding that Jev's Choice probabilities are systematically distorted and that inverting a latent third truth value of uncertainty estimated from Score expectations raises median Choice soft accuracy from 0.771 to 0.978.
Porcedda built Sys1Cal-v1, a synthetic dataset of 365 True/False items derived from 92 probability problems whose proposition probabilities are known by construction, and used total variation distance and distributional overlap (soft accuracy) to evaluate Jev's Noul, Choice, and Score primitives plus the open-source baseline SemIf, finding that Jev's Choice probabilities are systematically distorted and that inverting a latent third truth value of uncertainty estimated from Score expectations raises median Choice soft accuracy from 0.771 to 0.978.
Scuola Superiore Sant'Anna University of Pisa
Riccardo Porcedda（First author, Corresponding author）
Sys1Cal-v1 Probability Calibration Jev System One models
Research Square Sep 15, 2026
Cross-cohort evaluation of thyroid ultrasound deep learning across different outcome definitions: a duplicate-controlled benchmark
This preprint trained five backbones under duplicate-controlled image-level partitioning (387 thyroid ultrasound images and 1,332 fine-needle aspiration cytology blocks from 385 public cases, labelled by postoperative diagnosis) and applied the frozen models to TN3K (n = 1,228, dataset-provided benign/malignant labels) and to a DDTI endpoint derived from radiologist TI-RADS categories (n = 637), finding ultrasound ensemble AUROC 0.825 and ResNet-50 0.888 on TN3K versus 0.477 and 0.437 against the TI-RADS-derived endpoint, with a cytology benchmark AUROC of 0.986 (0.970–0.998) against 0.733 for the ultrasound ensemble, while noting that cohort, acquisition and endpoint changed together so the contrast cannot be attributed to label definition alone.
This preprint trained five backbones under duplicate-controlled image-level partitioning (387 thyroid ultrasound images and 1,332 fine-needle aspiration cytology blocks from 385 public cases, labelled by postoperative diagnosis) and applied the frozen models to TN3K (n = 1,228, dataset-provided benign/malignant labels) and to a DDTI endpoint derived from radiologist TI-RADS categories (n = 637), finding ultrasound ensemble AUROC 0.825 and ResNet-50 0.888 on TN3K versus 0.477 and 0.437 against the TI-RADS-derived endpoint, with a cytology benchmark AUROC of 0.986 (0.970–0.998) against 0.733 for the ultrasound ensemble, while noting that cohort, acquisition and endpoint changed together so the contrast cannot be attributed to label definition alone.
Central Asian University INTI International University University of Central Asia
Behnam Kiani Kalejahi（First author）
Cross-Cohort Evaluation Duplicate Control Probability Calibration External Validation Medical Image Classification

You’re all caught up

Teaching a model to say "I don't know": a separate answer selector beats retraining the policy, lifting TriviaQA coverage from 20.2% to 30.9%

Porcedda's Sys1Cal-v1 shows Jev's Choice probabilities are systematically distorted, and restoring an uncertainty component lifts median soft accuracy from 0.771 to 0.978

Cross-cohort evaluation of thyroid ultrasound deep learning across different outcome definitions: a duplicate-controlled benchmark