BF16 catches FP16 in RL post-training: stochastic rounding stops small weight updates from sticking at low learning rates
At the low learning rates common in reinforcement learning with verifiable rewards, mixed-precision BF16 training converges more slowly than FP16 because small updates to FP32 shadow weights are often erased when those weights are rounded to BF16, leaving the weights unchanged for several steps. Replacing round-to-nearest with stochastic rounding in that quantization step makes BF16 follow FP16's reward trajectory on 1.5B, 14B and 30B-A3B models across two math datasets, while drifting less from the reference model than BF16 at a raised learning rate and keeping BF16's dynamic range without loss scaling.