PPO's clip was only one geometry: swapping in KL divergence with atomic estimation lifts math average from 27.7 to 41.5
Synopsis
PPO's clipping in LLM reinforcement learning is generalized into a two-dimensional design space of divergence geometry and estimator topology, where KL divergence with atomic estimation raises average scores by about 50% on math and 14% on coding over standard GRPO across eight competition-level benchmarks.
Interpretation
Proximal policy optimization for large language models is no longer treated as one fixed clipping rule, but as a design space with two independently choosable dimensions: which divergence is constrained, and which estimator computes it. PPO and its recent variants each modified the trust region in fragmented ways motivated by specific empirical observations, with no unified first-principles account of how those mechanisms relate. Across eight competition-level math and coding benchmarks, a grid of four divergences and three estimators is evaluated, and PPO, DAPO, DPPO, and RIPO each fall on specific cells of it.
The constrained divergence sets how far a policy update may move, while the estimator sets how that constraint is actually enforced under sampling. PPO constrains a single-sample Monte Carlo estimate of total variation divergence, whose generating function has zero curvature and symmetric bounds, giving no adaptive resistance to the size of a policy shift. On mathematical reasoning, KL divergence with atomic estimation averages 41.5 against 27.7 for total variation with single-sample Monte Carlo; on code generation the same pair gives 59.7 against 52.4.
Atomic estimation weights each action's divergence contribution by its reference probability, so a large ratio shift on a low-probability action is not automatically treated as a violation. Single-sample Monte Carlo evaluates only the sampled action, making its clipping decisions highly sensitive to sampling variance. Atomic estimation ranks first in every column of the code table, and averaged over the four divergences it gains more than single-sample Monte Carlo on both math and code, while Monte Carlo entropy collapses rapidly toward zero and its score curves plateau early.
Perspective
The reported results cover training on Qwen3-8B-Base with DAPO-Math-17k and Eurus-Code-25k and evaluation on eight competition-level math and coding benchmarks with verifiable correct-or-wrong rewards. It is aimed at researchers and engineers designing or tuning proximal constraints for LLM reinforcement learning, to help decide which divergence geometry and estimator topology to use.
The ranking of divergence geometries and estimator topologies is not uniform across tasks: binary estimation reverses the order on math while giving only modest gains for non-total-variation divergences on code. The bias-variance advantage of atomic estimation over single-sample Monte Carlo comes from an appendix analysis, and its realized benefit still depends on the distribution of reference probabilities.