# RL Post-Training

11 related articles

Subscribe to updates

arXiv Nov 10, 2026
BF16 catches FP16 in RL post-training: stochastic rounding stops small weight updates from sticking at low learning rates
At the low learning rates common in reinforcement learning with verifiable rewards, mixed-precision BF16 training converges more slowly than FP16 because small updates to FP32 shadow weights are often erased when those weights are rounded to BF16, leaving the weights unchanged for several steps. Replacing round-to-nearest with stochastic rounding in that quantization step makes BF16 follow FP16's reward trajectory on 1.5B, 14B and 30B-A3B models across two math datasets, while drifting less from the reference model than BF16 at a raised learning rate and keeping BF16's dynamic range without loss scaling.
At the low learning rates common in reinforcement learning with verifiable rewards, mixed-precision BF16 training converges more slowly than FP16 because small updates to FP32 shadow weights are often erased when those weights are rounded to BF16, leaving the weights unchanged for several steps. Replacing round-to-nearest with stochastic rounding in that quantization step makes BF16 follow FP16's reward trajectory on 1.5B, 14B and 30B-A3B models across two math datasets, while drifting less from the reference model than BF16 at a raised learning rate and keeping BF16's dynamic range without loss scaling.
Mahdi Nazemi（First author）
Stochastic Rounding Weight Sticking RL Post-Training Reinforcement Learning with Verifiable Rewards
arXiv Nov 11, 2026
A fixed KL reference can collapse the odds of optimal answers: resetting it to the latest policy restores global convergence for PPO and GRPO variants
Policy optimization in language-model post-training starts from a capable base model, yet REINFORCE, the basis of PPO and GRPO, can drive the probability of optimal responses toward zero even as reward improves, and convergence can take exponentially many steps even with exact gradients and KL regularization to the base model. Periodically resetting the KL reference to the most recent policy, formalized as Approximate Policy Mirror Descent (APMD), lets each subproblem be solved locally under strong regularization while successive resets accumulate into global optimality. For softmax linear policies, this yields the first global convergence and coverage-preserving guarantees for PPO and GRPO variants with reference resets and for Kimi-style squared-loss algorithms.
Policy optimization in language-model post-training starts from a capable base model, yet REINFORCE, the basis of PPO and GRPO, can drive the probability of optimal responses toward zero even as reward improves, and convergence can take exponentially many steps even with exact gradients and KL regularization to the base model. Periodically resetting the KL reference to the most recent policy, formalized as Approximate Policy Mirror Descent (APMD), lets each subproblem be solved locally under strong regularization while successive resets accumulate into global optimality. For softmax linear policies, this yields the first global convergence and coverage-preserving guarantees for PPO and GRPO variants with reference resets and for Kimi-style squared-loss algorithms.
Audrey Huang（First author）
Approximate Policy Mirror Descent Coverage Collapse KimiPO Proximal Policy Optimization RL Post-Training
vLLM Blog Sep 14, 2026
Training and rollout stop disagreeing: vime and RL-Kernel hold logprobs bitwise identical for 200 steps on AMD MI300X
In reinforcement-learning post-training, the training engine and the rollout engine can compute different logprobs even with the same model, weights and inputs, because kernels, parallelism and reduction order differ, which affects the importance ratio, KL diagnostics and clipping. vime aligns the training timeline while RL-Kernel aligns numerical execution, and together they put attention, feed-forward, logprob and communication paths on one numerical contract on AMD Instinct MI300X. In a Qwen3-8B GRPO experiment on 8 MI300X GPUs, the strict path ran 200 consecutive steps with mismatch_count and max_abs_diff both zero.
In reinforcement-learning post-training, the training engine and the rollout engine can compute different logprobs even with the same model, weights and inputs, because kernels, parallelism and reduction order differ, which affects the importance ratio, KL diagnostics and clipping. vime aligns the training timeline while RL-Kernel aligns numerical execution, and together they put attention, feed-forward, logprob and communication paths on one numerical contract on AMD Instinct MI300X. In a Qwen3-8B GRPO experiment on 8 MI300X GPUs, the strict path ran 200 consecutive steps with mismatch_count and max_abs_diff both zero.
RL Post-Training RL-Kernel vime Bitwise train-rollout consistency
arXiv Nov 6, 2026
Reasoning-model RL training stops tuning only execution: RPEG also searches pipeline structure, up to 2x faster than ReaL on 256 GH200 GPUs
Reinforcement-learning training of reasoning language models chains generation, assessment and training across several large models each step, and existing systems usually optimize only part of that design space. RPEG places pipeline structure, including bounded asynchrony and actor–critic sharing, and execution choices such as stage fusion, fine-grained scheduling and multi-dimensional parallelism into one graph searched jointly, backed by a hardware-calibrated analytical performance model. On up to 256 GH200 GPUs its runtime is up to 2x faster end to end than ReaL, cuts step latency by up to 2.48x versus verl, and raises hardware utilization by up to 30%.
Reinforcement-learning training of reasoning language models chains generation, assessment and training across several large models each step, and existing systems usually optimize only part of that design space. RPEG places pipeline structure, including bounded asynchrony and actor–critic sharing, and execution choices such as stage fusion, fine-grained scheduling and multi-dimensional parallelism into one graph searched jointly, backed by a hardware-calibrated analytical performance model. On up to 256 GH200 GPUs its runtime is up to 2x faster end to end than ReaL, cuts step latency by up to 2.48x versus verl, and raises hardware utilization by up to 30%.
Maciej Besta（First author）
RLM Pipeline Execution Graph (RPEG)RL Post-Training Asynchronous RL Simulated Annealing
NVIDIA Technical Blog Nov 7, 2026
RL weight refit stops broadcasting whole tensors: NCCL M2N Reshard ships open source, cutting refit latency by up to about five times
In reinforcement learning post-training, training and inference run on separate GPUs, and the latest weights must be moved from the training side to the inference side at every step; this transfer is called refit. The conventional approach all-gathers the complete parameter and then broadcasts it, so an inference GPU receives the full tensor even when it needs only a shard, giving a payload of roughly twice the model's weights. NVIDIA recasts refit as resharding a distributed tensor between the training and inference layouts and releases NCCL M2N Reshard as an open-source reference implementation; in NeMo RL benchmarks, refit latency falls by 2.7x to 5.7x relative to gather-and-broadcast refit.
In reinforcement learning post-training, training and inference run on separate GPUs, and the latest weights must be moved from the training side to the inference side at every step; this transfer is called refit. The conventional approach all-gathers the complete parameter and then broadcasts it, so an inference GPU receives the full tensor even when it needs only a shard, giving a payload of roughly twice the model's weights. NVIDIA recasts refit as resharding a distributed tensor between the training and inference layouts and releases NCCL M2N Reshard as an open-source reference implementation; in NeMo RL benchmarks, refit latency falls by 2.7x to 5.7x relative to gather-and-broadcast refit.
NVIDIA
Reshard Collective NCCL M2N Reshard Weight Refit RL Post-Training
arXiv Oct 1, 2026
LSD cuts the length-scaling tax on already-solved queries from 19.0% to -3.7% while matching or beating RL Pass@1
The work defines the length-scaling tax (LST) as the excess response length that RL post-training adds on already-solved queries without a matching accuracy gain, and proposes Length Self-Distillation (LSD), which routes solved prompt groups to on-policy distillation while keeping the original RLVR objective for unsolved groups, using an exponential moving average of the same policy lineage as teacher with no external model; LSD reduces LST from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks while matching or improving average Pass@1 over RL.
The work defines the length-scaling tax (LST) as the excess response length that RL post-training adds on already-solved queries without a matching accuracy gain, and proposes Length Self-Distillation (LSD), which routes solved prompt groups to on-policy distillation while keeping the original RLVR objective for unsolved groups, using an exponential moving average of the same policy lineage as teacher with no external model; LSD reduces LST from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks while matching or improving average Pass@1 over RL.
Peking University Tongji University
Xu Wan（First author）
Length Self-Distillation Length-Scaling Tax On-Policy Distillation RL Post-Training
arXiv Oct 1, 2026
RL post-training of vision-language-action models concentrates parameter updates in Timestep Modules that hold only 14.83%–27.58% of the action expert, and their low-rank directions predict and improve task success
This work systematically analyzes how reinforcement learning (RL) reshapes flow-based vision-language-action (VLA) models, finding that across π0.5 and GR00T N1.5/N1.6 on LIBERO, ManiSkill, MetaWorld, and CALVIN, RL induces low-rank parameter updates highly concentrated in the action expert's Timestep Modules, which hold 14.83%–27.58% of the parameters yet capture a disproportionate share of the RL gain, with shift-vector update directions predicting task success at up to 99.6% ROC-AUC and steering along them improving policies without additional RL training.
This work systematically analyzes how reinforcement learning (RL) reshapes flow-based vision-language-action (VLA) models, finding that across π0.5 and GR00T N1.5/N1.6 on LIBERO, ManiSkill, MetaWorld, and CALVIN, RL induces low-rank parameter updates highly concentrated in the action expert's Timestep Modules, which hold 14.83%–27.58% of the parameters yet capture a disproportionate share of the RL gain, with shift-vector update directions predicting task success at up to 99.6% ROC-AUC and steering along them improving policies without additional RL training.
Minjae Oh（First author）
Vision-Language-Action model RL Post-Training Low-Rank Parameter Updates Timestep Modules Activation Steering
arXiv Sep 30, 2026
SoL-Refiner refines low-resolution generated video to 4K in one denoising step, cutting 2K refinement latency from 57.461 to 6.447 seconds
The work presents SoL-Refiner, a one-step video refiner that turns low-resolution generator outputs into 4K video through a three-stage recipe of high-resolution continual training, reinforcement-learning post-training with frame-based reward models, and final one-step distillation, and introduces Refiner-Bench, a video refinement benchmark using a shared-input protocol to compare refiners at roughly 2K output resolution; at 2K the one-step model outperforms all evaluated external refiners on VBench and UniPercept averages, at 3840×2176 it improves both metrics over the three-step LTX-2.3 Refiner, and with the complete acceleration stack it achieves an 8.91× speedup in refinement latency over the same baseline in the 2K latency setting.
The work presents SoL-Refiner, a one-step video refiner that turns low-resolution generator outputs into 4K video through a three-stage recipe of high-resolution continual training, reinforcement-learning post-training with frame-based reward models, and final one-step distillation, and introduces Refiner-Bench, a video refinement benchmark using a shared-input protocol to compare refiners at roughly 2K output resolution; at 2K the one-step model outperforms all evaluated external refiners on VBench and UniPercept averages, at 3840×2176 it improves both metrics over the three-step LTX-2.3 Refiner, and with the complete acceleration stack it achieves an 8.91× speedup in refinement latency over the same baseline in the 2K latency setting.
Haozhe Liu（First author）
SoL-Refiner Video Refinement RL Post-Training Distribution Matching Distillation Refiner-Bench
arXiv Sep 29, 2026
Nereus adapts RL post-training parallelism on the fly: 27.7% lower average step latency on a real-data trace and up to 7.27x OpenRLHF throughput for 8B PPO
Nereus is a cost-aware runtime for reinforcement-learning post-training of large language models that represents each model-stage replica as an Elastic Model Unit and orchestrates Split, Merge, Extend, and Destroy primitives through a global transition graph, adapting the DP/TP/PP execution plan online under a payback rule; on a trace built from real data, online TP/PP adaptation cuts average step latency by 27.7% versus the initial fixed TP/PP layout, six transitions in a 1,000-step run scaling to 1,024 GPUs consume 0.079% of total run time, and end-to-end 8B PPO throughput improves by 2.14-7.27x over OpenRLHF and 1.10-1.47x over Verl.
Nereus is a cost-aware runtime for reinforcement-learning post-training of large language models that represents each model-stage replica as an Elastic Model Unit and orchestrates Split, Merge, Extend, and Destroy primitives through a global transition graph, adapting the DP/TP/PP execution plan online under a payback rule; on a trace built from real data, online TP/PP adaptation cuts average step latency by 27.7% versus the initial fixed TP/PP layout, six transitions in a 1,000-step run scaling to 1,024 GPUs consume 0.079% of total run time, and end-to-end 8B PPO throughput improves by 2.14-7.27x over OpenRLHF and 1.10-1.47x over Verl.
Zhejiang University Aalto University Shenzhen University of Advanced Technology
Songlin Jiang（First author）
Nereus Adaptive Parallelism Elastic Model Unit RL Post-Training
arXiv Sep 29, 2026
QwenGyre pairs elastic GPU scheduling with trajectory-tree processing to lift Qwen 3.8 2.4T from 52.5% to 58.5% on NL2RepoBench in 48 steps
QwenGyre is an end-to-end framework for extreme-long-horizon (xLong) online agent RL that elastically reallocates GPUs between rollout and training without interrupting live harness executions, while its trajectory processor reconstructs branching executions into shared-prefix trajectory trees, scores partial progress after timeouts, and deduplicates sampled trajectories by role priority; scaled to Qwen 3.8 2.4T with 700K tokens per rollout, it raises NL2RepoBench pass rate from 52.5% to 58.5% in 48 steps and delivers up to 1.85x and 1.78x end-to-end speedups over Colocate and Async.
QwenGyre is an end-to-end framework for extreme-long-horizon (xLong) online agent RL that elastically reallocates GPUs between rollout and training without interrupting live harness executions, while its trajectory processor reconstructs branching executions into shared-prefix trajectory trees, scores partial progress after timeouts, and deduplicates sampled trajectories by role priority; scaled to Qwen 3.8 2.4T with 700K tokens per rollout, it raises NL2RepoBench pass rate from 52.5% to 58.5% in 48 steps and delivers up to 1.85x and 1.78x end-to-end speedups over Colocate and Async.
Qwen Alibaba Group (China)Tsinghua University
Weiqi Wang（First author）
QwenGyre Agentic Reinforcement Learning Trajectory Processing RL Post-Training long-horizon software engineering