# Trajectory-Aware Policy Optimization

In long-horizon autoregressive reinforcement learning, local policy mismatch compounds along the trajectory into state distribution shift, destabilizing training and eventually collapsing it as the horizon grows. Trajectory-Aware Policy Optimization derives a state-distribution correction from the off-policy policy gradient and writes it into the objective, estimates it with a geometric mean to cut variance, and tightens the clipping boundary with the length of the preceding trajectory. Across eight competition-level math and code benchmarks it improves over GRPO by up to 48% on math and 12% on code on average, with training scores rising steadily while KL divergence stays controlled.

In long-horizon autoregressive reinforcement learning, local policy mismatch compounds along the trajectory into state distribution shift, destabilizing training and eventually collapsing it as the horizon grows. Trajectory-Aware Policy Optimization derives a state-distribution correction from the off-policy policy gradient and writes it into the objective, estimates it with a geometric mean to cut variance, and tightens the clipping boundary with the length of the preceding trajectory. Across eight competition-level math and code benchmarks it improves over GRPO by up to 48% on math and 12% on code on average, with training scores rising steadily while KL divergence stays controlled.