RLCPR characterizes posterior concentration and beats a state-of-the-art verifier-free RL baseline by up to 4.0% on six of seven benchmarks

The paper identifies and names a length-dependent failure mode of probability rewards, the Posterior Concentration Phenomenon (PCP): as a reasoning trace becomes lengthy, the probability of a reference answer conditioned on that trace often collapses to a low-variance interval. The reliability of probability-based rewards in verifier-free RL, especially in long-horizon reasoning, had remained underexplored; this work characterizes the phenomenon explicitly as a trace-length-dependent failure mode. Based on observational analysis of how the conditional probability varies with trace length, described in the abstract as often collapsing to a low-variance interval.

The paper argues that PCP yields nearly indistinguishable rewards, which under GRPO-based settings makes probability-based policy optimization unstable and inefficient. It links reward-level collapse to optimization-level instability and inefficiency, indicating the phenomenon is not merely a measurement issue but affects training. The abstract states this mechanism-level claim, scoped to GRPO-based settings.

The paper proposes RLCPR, a verifier-free RL framework that explicitly accounts for PCP, with two components: uncertainty-aware data sampling and concentration-aware regularization. Uncertainty-aware data sampling reduces concentration-prone rollouts before generation, while concentration-aware regularization penalizes unnecessarily long traces when posterior rewards collapse. The two components correspond to pre-generation and in-training interventions, with their roles stated in the abstract.

Experiments show RLCPR, alongside higher token efficiency, outperforms the state-of-the-art verifier-free RL baseline by up to 4.0% on six of seven benchmarks, including general-domain and mathematical reasoning challenges. Relative to the baseline, the framework improves on most benchmarks while also improving token efficiency. The abstract reports up to 4.0% and six of seven benchmarks, noting the benchmarks span general-domain and mathematical reasoning challenges.

The work targets reinforcement learning settings without external verifiers that use probability as reward, particularly long-horizon reasoning training under GRPO-style setups, aiming to improve optimization stability and token efficiency. For LLM reasoning researchers and practitioners using such rewards, RLCPR offers two directly reusable intervention points: sampling that reduces concentration-prone rollouts before generation, and regularization that penalizes overly long traces when rewards collapse. Applicability is scoped by the abstract to general-domain and mathematical reasoning benchmarks.

The visible text is abstract-level, without benchmark names, model scale, training configuration, or statistical significance, so the conditions behind the 4.0% gain and the token-efficiency improvement still need checking in the full text. How PCP is thresholded, how the low-variance interval is defined, and the separate contribution of each component are also questions a reader would keep in view. Whether the phenomenon appears under other reward forms or non-GRPO optimizers is a direction worth watching.