Async RL stops being dragged down by kernel mismatch: REINFORCE Zero keeps learning at policy lag 64 and reaches 2.40× synchronous training-step throughput
Synopsis
Asynchronous reinforcement learning runs rollout and training at the same time, but the learner trains on samples from older weights, and importance sampling alone had not stopped the drift. REINFORCE Zero, from NVIDIA Research and Inferact, first aligns the rollout engine and trainer so both compute bit-identical per-token logprobs at the same weights, then handles the remaining weight lag with unclipped token-level importance sampling. On 16 GPUs over 12K updates it reaches 2.40× the training-step throughput of synchronous training with aligned kernels (68.55 versus 164.77 hours) at comparable quality, and Qwen3-4B with Python tools trains at lag 64 through 5,000 updates, with mean reward rising from 0.287 to about 0.78.
Interpretation
Asynchronous reinforcement learning can train stably without sample-rejection gates, using kernel alignment plus unclipped importance sampling. Previously the gap between rollout and learner in asynchronous training had two parts: weight lag, which changes smoothly across updates, and kernel mismatch, where the rollout engine and trainer compute different numbers for the same weights and which fluctuates with batch composition and numerical execution, persisting even under importance sampling. On DeepSeek-R1-Distill-Qwen-1.5B, the peak-to-median ratio of the importance-weight deviation D3 is 1.50× with aligned kernels and 442.54× with standard kernels; standard-kernel asynchronous importance sampling raises rollout-to-learner KL 83.33× between the first and last 1,000 updates and drops online AIME24 pass@1 from 34.27% to 13.96%, while aligned kernels stay flat.
The recipe has three parts: make rollout and training produce bit-identical logprobs for the same weights and tokens, reweight each token by the recorded sampling probability without clipping, and optimize plain REINFORCE with no value model. Importance sampling weights each token by the learner's probability divided by the probability recorded at sampling time, and clipping generally introduces bias, so the raw ratio is kept with no filtering. On the 1.5B reasoning task at 6,000 updates, REINFORCE Zero at lag 2 and lag 8 reaches macro pass@1 of 48.80% and 48.53%, synchronous training reaches 48.91%, and standard-kernel asynchronous importance sampling reaches 45.32%, 46.79% with a sequence gate and 44.59% with a token gate.
Alignment has a cost, but it buys long-run stability and continued learning at high lag. Aligned kernels are slower than the fastest standard kernels: on OSWorld, REINFORCE Zero takes 944.25 s per step on average versus 884.17 s for standard-kernel importance sampling with a sequence gate, 6.8% more. Qwen3-4B with Python tools and a 16K context trains at policy lag 64 through 5,000 updates, with mean reward rising from 0.287 in the first 100 updates to about 0.78 and AIME24/25/26 pass@1 improving over the same run; Qwen3-30B-A3B raises macro pass@1 from 31.29% to 71.55% in 3,000 updates.
Perspective
The demonstrated scope covers stable training through 20,000 updates on a 1.5B reasoning model, 5,000 updates at lag 64 with Qwen3-4B and Python tools, tool use with Qwen3-30B-A3B, and the OSWorld computer-use task; DeepSeek-V4.1-Flash and Kimi K3.1 have so far only been validated for alignment, with long asynchronous runs still to come.
Performance optimization for Kimi K3.1 is ongoing, and its cost against the fast baseline on AMD MI355X is still being measured; long asynchronous training on DeepSeek-V4.1-Flash and Kimi K3.1 is listed as the next step, with no results yet.