Confidence estimators drift after agentic RL: StaR-TTA reorders them by trajectory stability, reaching 0.7208 average AUROC on four QA benchmarks
Synopsis
Agentic reinforcement learning shifts the reasoning trajectories of large language models, and trajectory-based confidence estimators trained on a base model lose much of their ranking ability after that shift. StaR-TTA treats trajectory stability as a label-free reliability signal: it learns the link between stability and correctness on labeled base trajectories, then uses stability rankings from unlabeled RL trajectories to correct confidence at test time. Across four QA benchmarks it reaches an average AUROC of 0.7208, with gains on unseen datasets, different model families and scales, and both PPO and GRPO.
Interpretation
After agentic reinforcement learning, confidence estimators that rely on reasoning trajectories lose much of their ranking ability, and changes in the trajectories themselves are a major contributor. On NQ, MICE-LR falls from 0.741 AUROC on the base model to 0.567 on the RL model, MICE-RF from 0.773 to 0.559, and CCPS from 0.745 to 0.491; DACA, which depends less directly on trajectory representations, stays nearly flat, from 0.659 to 0.656. With tool results and final answers held fixed and only the RL trajectory replaced by its base-model counterpart, MICE-LR, MICE-RF and CCPS recover 44.5%, 51.9% and 11.1% of the remaining AUROC loss, respectively.
Trajectory stability keeps predicting answer correctness before and after RL, so it can serve as a reliability ordering when correctness labels are unavailable. Each trajectory is fixed at the prefix before final-answer generation and only five short answers are resampled; stability is described by answer support, distribution concentration and the frequency gap between the two most common answers, and these signals score above 0.5 AUROC on both base and RL models, with the equal-weight average around 0.7 to 0.8. StaR-TTA jointly learns correctness supervision and stability-derived reliability supervision on labeled base trajectories, and at test time updates only a linear corrector conditioned on the trajectory representation while both language models, the stability-to-reliability mapper, the confidence scorer and the calibrator stay frozen.
Across four QA benchmarks, StaR-TTA reaches an average AUROC of 0.7208, average ECE of 0.0688, average Brier score of 0.1870 and average NLL of 0.5729 on RL trajectories. On MuSiQue, HTC reaches 0.721 AUROC, above StaR-TTA's 0.672; on TriviaQA, HotpotQA and PopQA, StaR-TTA reaches 0.800, 0.715 and 0.696 AUROC, respectively. Trained on TriviaQA and tested on 2WikiMultiHopQA, Bamboogle and NQ, StaR-TTA has the highest AUROC and the lowest Brier score and NLL on all four datasets; on Llama-3.2-3B-Instruct its AUROC is 5.6 to 7.8 points above the strongest baseline, and on Qwen2.5-7B-Instruct 1.1 to 10.2 points above.
Perspective
The demonstrated setting is Qwen2.5-3B with Search-R1 agentic RL, evaluated on TriviaQA, HotpotQA, MuSiQue and PopQA and extended to Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct, and the PPO and GRPO algorithms; test-time adaptation updates only a linear corrector, and stability is estimated by fixing the prefix and resampling five short answers.
With very few adaptation samples the results are unstable or even worse, with average AUROC at 0.6306 for 2 samples and 0.6680 for 4, recovering to 0.7155 at 64 samples and 0.7208 with all samples; removing test-time adaptation gives an average AUROC of 0.7035, so adaptation mainly improves ranking while average ECE rises slightly from 0.0654 to 0.0688.