# Coverage Collapse

Policy optimization in language-model post-training starts from a capable base model, yet REINFORCE, the basis of PPO and GRPO, can drive the probability of optimal responses toward zero even as reward improves, and convergence can take exponentially many steps even with exact gradients and KL regularization to the base model. Periodically resetting the KL reference to the most recent policy, formalized as Approximate Policy Mirror Descent (APMD), lets each subproblem be solved locally under strong regularization while successive resets accumulate into global optimality. For softmax linear policies, this yields the first global convergence and coverage-preserving guarantees for PPO and GRPO variants with reference resets and for Kimi-style squared-loss algorithms.

Policy optimization in language-model post-training starts from a capable base model, yet REINFORCE, the basis of PPO and GRPO, can drive the probability of optimal responses toward zero even as reward improves, and convergence can take exponentially many steps even with exact gradients and KL regularization to the base model. Periodically resetting the KL reference to the most recent policy, formalized as Approximate Policy Mirror Descent (APMD), lets each subproblem be solved locally under strong regularization while successive resets accumulate into global optimality. For softmax linear policies, this yields the first global convergence and coverage-preserving guarantees for PPO and GRPO variants with reference resets and for Kimi-style squared-loss algorithms.