Chain-of-thought need not fire on every query: confidence gating cuts reasoning use by 30% without losing accuracy when the signal is strong

Having a model answer directly with a confidence score and enabling chain-of-thought only when that score falls below a threshold can save substantial reasoning cost on some models, with no training required. Earlier when-to-think methods mostly relied on reinforcement learning or classifiers and needed extra training, while training-free alternatives focused solely on perplexity, and whether confidence signals could serve as a reliable switch was unclear. On GPT-OSS-20B and Qwen3-32B, signals such as margin sampling, perplexity or P(True) clearly beat random gating, matching always-on chain-of-thought accuracy while reducing chain-of-thought usage by roughly 30% and saving an average of 70 to 100 tokens per query.

Whether gating works depends on whether the confidence signal separates correct from incorrect queries, not on how well its absolute values are calibrated. Intuition suggests a signal with low calibration error should be more reliable, yet GPT-5.1's verbalised confidence and GPT-OSS-20B's margin sampling have nearly identical calibration error and very different gating performance. GPT-OSS-20B margin reaches an AUROC of 0.798 against 0.604 for GPT-5.1 verbalised confidence, and the former gates far better; on Qwen3-8B and GPT-5.1 no tested confidence method consistently beats random gating across all budgets.

Pareto-optimal threshold selection picks the lowest-token threshold whose accuracy stays within a tolerance, so strong-signal combinations genuinely save compute while weak-signal combinations safely default toward full chain-of-thought. The prior confidence-routing method CAR fits a Gaussian to perplexity to derive thresholds and loses 7.3% accuracy on Qwen3-32B with its best signal, while a supervised MLP probe loses between 2.5% and 4.6% across models. With a 1% accuracy tolerance, a 10% calibration split and 100 Monte Carlo repeats, Qwen3-32B with P(True) saves an average of 171 tokens per query, a 26% cut in chain-of-thought usage translating to 23% lower latency; GPT-OSS-20B saves about 65 tokens via margin or perplexity with up to 20% latency reduction; GPT-5.1 saves only 1.5 tokens as the threshold defaults toward full chain-of-thought.

The findings rest on four training-free confidence signals, four models (Qwen3-8B, Qwen3-32B, GPT-OSS-20B and GPT-5.1) and seven benchmarks spanning commonsense, knowledge, mathematical and scientific, and soft reasoning; Pareto thresholds require a labelled calibration set containing both direct and chain-of-thought outputs. The authors note that the calibration set should reflect the difficulty and diversity of the target tasks, and that a broad, moderately difficult held-out set is sufficient for reliable threshold transfer.

The authors state that verbalised confidence uses a single standard prompting strategy, leaving fewer-sample or trained confidence estimators to future work; comparisons with fully trained when-to-think models and with conformal risk control are also left out, so the reported outperformance is relative only to the CAR and MLP-probe baselines evaluated. The qualitative analysis shows Excess CoT is the largest share for most models, meaning the direct answer was already correct but confidence was too low, so further savings are limited by models under-reporting confidence on correct answers.