Samplers no longer match the trainer: mixed-precision, mixed-accelerator RL trains stably and costs about 15% less
Synopsis
In large-model reinforcement learning, samplers have usually run on the same accelerator generation and precision as the trainer to keep the two numerically close. Xiaomi's MiMo team instead spread samplers for Qwen3.6-35B-A3B agentic coding across six accelerators, B200, B300, H100, H200, A100 and RTX PRO 6000, each with its own NVFP4, FP8 or BF16 weights and kernels, while the trainer updates in BF16 on B200. With off-policy correction and autoscaling serverless GPUs, the mixed samplers match homogeneous samplers on SWE-bench Pro V2 and cut sampling cost per update by about 15% for most of training.
Interpretation
Samplers can be decoupled from the trainer's accelerator generation and precision without losing stability or score. Large-model RL has usually kept samplers numerically close to the trainer, and sampling already takes 80–85% of the GPUs in large RL runs. On SWE-bench Pro V2, mixed and homogeneous samplers stay close in pass@4 throughout training; the mixed samplers' sampler–trainer KL averages 4.7 times the homogeneous one and training remains stable.
Stability comes from off-policy correction rather than from aligning the two sides numerically. Without correction, GRPO collapses soon after training starts on the mixed samplers, and on the homogeneous samplers training lasts much longer before ultimately collapsing too. Only SC + MIS, with or without top-p, trains stably to the end: pass@4 on HARD-51 rises over roughly the first 100–140 steps and levels off at about 65–75%, while the full set goes from 89% to about 94%.
The correction has three parts: masked importance sampling, score centering and top-p mask replay. Each piece on its own only delays the collapse, and every such run ends at or below the base model. Masked importance sampling weights each token by its importance ratio and masks tokens whose ratio falls outside a fixed range; score centering subtracts the expected gradient of the log-probability under the sampling policy to remove drift; top-p lets samplers sample only from the candidate set and the trainer renormalizes over the same set.
Decoupling samplers from the trainer makes the same results cheaper to reach. Homogeneous samplers share the trainer's accelerator and precision, so only kernels and staleness separate them. For most of training, the samplers' cost per update was about 15% lower on the mixed samplers than on the homogeneous samplers; the setup was built to validate performance rather than minimal cost, so more savings remain.
Perspective
The demonstrated setting is fully asynchronous training of Qwen3.6-35B-A3B on agentic coding tasks, with contexts up to 256K tokens and an average staleness of about 4, evaluated on SWE-bench Pro V2 and its HARD-51 subset with every checkpoint evaluated in BF16. The intended readers are engineers and researchers building large-scale RL training and inference systems who need sampling and training to be separable. The authors' stated next directions include sizing the sampling fleet separately from the trainer and growing it mid-run, carrying inference optimizations such as speculative decoding, lower precision, and split prefill and decode straight into RL, and moving the trainer to its own low precision.
The corrections work one token at a time, so prefixes and advantages still come from the samplers, and sequence-level corrections and staleness bounds are not used; top-p mask replay means the update optimizes the policy truncated to the top-p candidate set, with no gradient for logits outside it at that position, while evaluation samples from the full vocabulary; correction leaves the samples as they are, so choosing a precision or accelerator remains a trade between throughput and sample efficiency. The authors say better corrections are likely to follow.