Zhicheng Cai

PPO's clipping in LLM reinforcement learning is generalized into a two-dimensional design space of divergence geometry and estimator topology, where KL divergence with atomic estimation raises average scores by about 50% on math and 14% on coding over standard GRPO across eight competition-level benchmarks.

PPO's clipping in LLM reinforcement learning is generalized into a two-dimensional design space of divergence geometry and estimator topology, where KL divergence with atomic estimation raises average scores by about 50% on math and 14% on coding over standard GRPO across eight competition-level benchmarks.

In long-horizon autoregressive reinforcement learning, local policy mismatch compounds along the trajectory into state distribution shift, destabilizing training and eventually collapsing it as the horizon grows. Trajectory-Aware Policy Optimization derives a state-distribution correction from the off-policy policy gradient and writes it into the objective, estimates it with a geometric mean to cut variance, and tightens the clipping boundary with the length of the preceding trajectory. Across eight competition-level math and code benchmarks it improves over GRPO by up to 48% on math and 12% on code on average, with training scores rising steadily while KL divergence stays controlled.

In long-horizon autoregressive reinforcement learning, local policy mismatch compounds along the trajectory into state distribution shift, destabilizing training and eventually collapsing it as the horizon grows. Trajectory-Aware Policy Optimization derives a state-distribution correction from the off-policy policy gradient and writes it into the objective, estimates it with a geometric mean to cut variance, and tightens the clipping boundary with the length of the preceding trajectory. Across eight competition-level math and code benchmarks it improves over GRPO by up to 48% on math and 12% on code on average, with training scores rising steadily while KL divergence stays controlled.