# REINFORCE Zero

Asynchronous reinforcement learning runs rollout and training at the same time, but the learner trains on samples from older weights, and importance sampling alone had not stopped the drift. REINFORCE Zero, from NVIDIA Research and Inferact, first aligns the rollout engine and trainer so both compute bit-identical per-token logprobs at the same weights, then handles the remaining weight lag with unclipped token-level importance sampling. On 16 GPUs over 12K updates it reaches 2.40× the training-step throughput of synchronous training with aligned kernels (68.55 versus 164.77 hours) at comparable quality, and Qwen3-4B with Python tools trains at lag 64 through 5,000 updates, with mean reward rising from 0.287 to about 0.78.

Asynchronous reinforcement learning runs rollout and training at the same time, but the learner trains on samples from older weights, and importance sampling alone had not stopped the drift. REINFORCE Zero, from NVIDIA Research and Inferact, first aligns the rollout engine and trainer so both compute bit-identical per-token logprobs at the same weights, then handles the remaining weight lag with unclipped token-level importance sampling. On 16 GPUs over 12K updates it reaches 2.40× the training-step throughput of synchronous training with aligned kernels (68.55 versus 164.77 hours) at comparable quality, and Qwen3-4B with Python tools trains at lag 64 through 5,000 updates, with mean reward rising from 0.287 to about 0.78.