PyTorch 2.15 makes its rewritten NCCL backend the default, and a hang now dumps every rank's state to disk
Synopsis
PyTorch 2.15 makes the rewritten NCCL backend, previewed as opt-in nccl2 in 2.14, the default for every nccl process group, keeping TORCH_DIST_USE_NCCL2=0 and the nccl-legacy name as rollback paths. At cluster scale a single stalled GPU could halt a whole training run and the cause was often reproduced by hand; a backend-agnostic flight recorder now writes a post-mortem to disk when a collective hangs, and a timeout on any rank triggers dumps from every rank.
Interpretation
When a model trains across many GPUs on NVIDIA hardware, the GPUs exchange data through NCCL, and in PyTorch 2.15 every job that uses nccl runs on the rewritten backend by default. PyTorch 2.14 only previewed the rewritten backend, ported from torchcomms, as the opt-in nccl2 backend; 2.15 makes it the default for every nccl process group. Rollback paths remain TORCH_DIST_USE_NCCL2=0 and the nccl-legacy backend name; communicators are created eagerly with the process group, and reconfiguration can opt into a shrink/grow fast path requiring NCCL 2.29 or newer so surviving ranks skip a full bootstrap.
To make the backend debuggable in production, a backend-agnostic flight recorder writes a post-mortem to disk when a collective hangs. At cluster scale a single stalled GPU could halt an entire training run, and finding the cause has often meant reproducing the failure by hand. Watchdog health is published to the debug server, so a timeout on any rank triggers flight-recorder dumps from every rank, capturing what each rank was doing at the moment of the hang.
The same release changes memory and communication control for distributed training: FSDP2 can keep precision-sensitive parameters in fp32, and pipeline parallelism allocates receive buffers just before each receive. FSDP2 previously had no per-parameter precision override of this kind, and pipelining allocated receive buffers for every microbatch up front. Parameters such as MoE router gates and norm weights can stay in fp32 through MixedPrecisionPolicy(param_dtype_override_fn=...) while the rest trains in bf16, still with one all-gather per fully_shard, and a full gradient-accumulation step can be captured in a single CUDA graph.
Perspective
The demonstrated scope is the default backend switch, rollback paths and flight-recorder behavior described in the PyTorch 2.15 release notes, for engineers running distributed training on NVIDIA or ROCm clusters; the notes give no controlled measurement of throughput or time to diagnose a hang.
The flight recorder is marked API Unstable, and the notes report neither its hit rate on real cluster failures nor its effect on training performance; the shrink/grow fast path requires NCCL 2.29 or newer, and reconfiguration for fault tolerance on ROCm requires RCCL 2.30.7.