RL weight refit stops broadcasting whole tensors: NCCL M2N Reshard ships open source, cutting refit latency by up to about five times
Synopsis
In reinforcement learning post-training, training and inference run on separate GPUs, and the latest weights must be moved from the training side to the inference side at every step; this transfer is called refit. The conventional approach all-gathers the complete parameter and then broadcasts it, so an inference GPU receives the full tensor even when it needs only a shard, giving a payload of roughly twice the model's weights. NVIDIA recasts refit as resharding a distributed tensor between the training and inference layouts and releases NCCL M2N Reshard as an open-source reference implementation; in NeMo RL benchmarks, refit latency falls by 2.7x to 5.7x relative to gather-and-broadcast refit.
Interpretation
Weight refit moves from gathering the complete parameter and broadcasting it to transferring the intersections of source and destination shards directly, so each inference GPU receives only the weight it needs. Existing refit implementations commonly use gather-and-broadcast: training GPUs all-gather their local shards into the complete parameter, broadcast it to inference GPUs, and each inference GPU keeps only the shard its local layout requires. At BF16 weights, Kimi K3's 2.8 trillion parameters amount to about 5.6 TB and Qwen 3.8 2.4T A95B's 2.4 trillion parameters to about 4.8 TB; counting both the all-gather and broadcast stages, one refit carries a logical payload of roughly twice the model's weights, about 11.2 TB and 9.6 TB respectively.
Refit is expressed as a single distributed-tensor reshard: given a source layout and a destination layout, the communication library derives which region of which source GPU goes to which destination GPU. Training and inference use different data, context, tensor, expert and pipeline parallelism, so shard boundaries do not line up, a destination shard may be assembled from portions of several source shards, and replication can make the same weights available from multiple source GPUs or needed by multiple destination GPUs. NCCL M2N Reshard first derives the exact intersections between source and destination shards, then maps those transfers onto the physical GPU network topology; replicas within one NVLink domain split the incoming data as evenly as possible and exchange the rest over NVLink, with network transfer and NVLink replication pipelined to hide each other's latency.
The NeMo RL reference implementation uses a hybrid path: feed-forward projection weights, about 95% of the model's parameters, take the reshard path, while other small weights keep the conventional gather-and-broadcast path. Handling every layer type and every engine's sharding scheme through one path creates many edge cases that are hard to maintain. On NVIDIA DGX H100 (NVL8) and DGX GB200 (NVL72), Qwen 3 235B refit from TP4 x PP8 x EP16 to TP8 x DP16 drops from 6.4 s to 1.9 s, DeepSeek v3 671B drops from 14.9 s to 2.6 s on H100 and from 6.1 s to 2.3 s on GB200, and Nemotron Ultra v3 drops from 8.2 s to 2.3 s; in a refit-only microbenchmark, Qwen3.8 2.4T BF16 took 4.3 s.
Perspective
The reported scope is latency of the refit step: benchmarks cover Qwen 3 235B, DeepSeek v3 671B and Nemotron Ultra v3 on NVIDIA DGX H100 (NVL8) and DGX GB200 (NVL72), plus a refit-only microbenchmark on Qwen3.8 2.4T BF16, and the intended readers are framework developers maintaining separate training and inference GPU pools in disaggregated RL post-training.
The hybrid path rests on feed-forward projection weights accounting for around 95% of parameters in the latest LLMs, with other small weights still on the conventional path. When GPU membership changes as nodes are added or removed, NeMo RL currently destroys and recreates the communication group, and resiliency support using the NCCL shrink and grow APIs is planned for future updates.