# Training Stability

The work identifies two critic failure modes that destabilize PPO for LLM reinforcement learning — filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, and heterogeneous return noise across prompts lets high-variance prompts dominate critic updates in finite batches — and introduces EasyPPO, which combines actor-only overlong filtering, noise-normalized critic regression weighting each prompt by the inverse standard deviation of its sampled returns, and moderately smaller critic mini-batches with gradient clipping, remaining stable across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24 and multi-turn search on Search-R1, with best validation scores improving over PPO by 14.89%, 2.

The work identifies two critic failure modes that destabilize PPO for LLM reinforcement learning — filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, and heterogeneous return noise across prompts lets high-variance prompts dominate critic updates in finite batches — and introduces EasyPPO, which combines actor-only overlong filtering, noise-normalized critic regression weighting each prompt by the inverse standard deviation of its sampled returns, and moderately smaller critic mini-batches with gradient clipping, remaining stable across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24 and multi-turn search on Search-R1, with best validation scores improving over PPO by 14.89%, 2.

This work studies the training stability of full-pipeline FP8 reinforcement learning for LLMs, identifies that compounded FP8 quantization noise distorts the importance ratio so that negative-advantage tokens are over-clipped at the lower bound and their gradients are erroneously zeroed, and proposes Calibrated Clipping, which dynamically realigns clipping bounds by matching the lower-bound clipping quantile of a BF16 reference and rebalancing the upper bound, eliminating entropy surges and restoring near-BF16 performance across GRPO and DAPO, 8B to 32B models, and multiple FP8 scaling granularities, while reaching up to 1.5x the BF16 training throughput.

This work studies the training stability of full-pipeline FP8 reinforcement learning for LLMs, identifies that compounded FP8 quantization noise distorts the importance ratio so that negative-advantage tokens are over-clipped at the lower bound and their gradients are erroneously zeroed, and proposes Calibrated Clipping, which dynamically realigns clipping bounds by matching the lower-bound clipping quantile of a BF16 reference and rebalancing the upper bound, eliminating entropy surges and restoring near-BF16 performance across GRPO and DAPO, 8B to 32B models, and multiple FP8 scaling granularities, while reaching up to 1.5x the BF16 training throughput.

The work proposes StableVQ, three parameter-free components — Dynamic STE, Region VQ Loss, and Decoupled Schedule — that disentangle Encoder–Decoder and Codebook training so each module can fulfill its own responsibility independently, consistently improving training stability, codebook utilization, and reconstruction quality on ImageNet across diverse codebook sizes and initialization settings.

The work proposes StableVQ, three parameter-free components — Dynamic STE, Region VQ Loss, and Decoupled Schedule — that disentangle Encoder–Decoder and Codebook training so each module can fulfill its own responsibility independently, consistently improving training stability, codebook utilization, and reconstruction quality on ImageNet across diverse codebook sizes and initialization settings.