Skip to main content

# pipeline parallelism

2 related articles

Subscribe to updates

PyTorch Oct 29, 2026
PyTorch 2.15 makes its rewritten NCCL backend the default, and a hang now dumps every rank's state to disk
PyTorch 2.15 makes the rewritten NCCL backend, previewed as opt-in nccl2 in 2.14, the default for every nccl process group, keeping TORCH_DIST_USE_NCCL2=0 and the nccl-legacy name as rollback paths. At cluster scale a single stalled GPU could halt a whole training run and the cause was often reproduced by hand; a backend-agnostic flight recorder now writes a post-mortem to disk when a collective hangs, and a timeout on any rank triggers dumps from every rank.
PyTorch 2.15 makes the rewritten NCCL backend, previewed as opt-in nccl2 in 2.14, the default for every nccl process group, keeping TORCH_DIST_USE_NCCL2=0 and the nccl-legacy name as rollback paths. At cluster scale a single stalled GPU could halt a whole training run and the cause was often reproduced by hand; a backend-agnostic flight recorder now writes a post-mortem to disk when a collective hangs, and a timeout on any rank triggers dumps from every rank.
PyTorch
PyTorch 2.15 rewritten NCCL backend FSDP2 pipeline parallelism NVGEMM for NVFP4 decode
arXiv Sep 29, 2026
FlashForward reuses in-flight KV cache with sparse clean anchors to speed 20s+ video generation by 1.16–1.69x on four GPUs while reaching 0.838 VBench Total
FlashForward publishes the in-flight KV cache already computed by each ordinary denoising forward for reuse by later chunks and complements it with sparse clean anchor KV generated ahead of time for long-range structural guidance, removing cache-update-only forwards; with up to four GPUs it runs 1.16–1.69x faster than HiAR and 1.42–2.92x faster than Self-Forcing for 16 FPS videos of 20 seconds or longer across 1.3B and 14B backbones at 480p and 720p, and the 1.3B model at 480p scores 0.838 on VBench-1.0 while remaining stable at 20s, 35s, and 65s.
FlashForward publishes the in-flight KV cache already computed by each ordinary denoising forward for reuse by later chunks and complements it with sparse clean anchor KV generated ahead of time for long-range structural guidance, removing cache-update-only forwards; with up to four GPUs it runs 1.16–1.69x faster than HiAR and 1.42–2.92x faster than Self-Forcing for 16 FPS videos of 20 seconds or longer across 1.3B and 14B backbones at 480p and 720p, and the 1.3B model at 480p scores 0.838 on VBench-1.0 while remaining stable at 20s, 35s, and 65s.
Yikai Wang（First author）
FlashForward KV Cache Autoregressive Video Generation pipeline parallelism

You’re all caught up