Confidence estimators drift after agentic RL: StaR-TTA reorders them by trajectory stability, reaching 0.7208 average AUROC on four QA benchmarks
Agentic reinforcement learning shifts the reasoning trajectories of large language models, and trajectory-based confidence estimators trained on a base model lose much of their ranking ability after that shift. StaR-TTA treats trajectory stability as a label-free reliability signal: it learns the link between stability and correctness on labeled base trajectories, then uses stability rankings from unlabeled RL trajectories to correct confidence at test time. Across four QA benchmarks it reaches an average AUROC of 0.7208, with gains on unseen datasets, different model families and scales, and both PPO and GRPO.