AI4S News · AI and science frontiers

Dropless MoE training in JAX gets a tenfold lift: DeepSeek-V3 671B holds 97% scaling efficiency at 1,024 GPUs

NVIDIA wired Transformer Engine's MoE optimizations into the JAX training stack so every token reaches its selected expert, with no dropping and no padding. An unoptimized baseline training DeepSeek-V3 on GB300 reached only 103 TFLOPS/GPU, with inter-GPU communication consuming 84% of accumulated kernel time; grouped GEMM, MXFP8 grouped quantization and fused expert-parallel dispatch and combine raised that to 1,068 TFLOPS/GPU, about a tenfold end-to-end throughput gain, and the stack sustained 97% efficiency at 1,024 GPUs.

PyTorch Oct 22, 2026

PyTorch on AMD GPUs gains a FlyDSL backend: kernels map work to hardware explicitly, and vLLM requests reach up to 3.01x

PyTorch users on AMD MI350-series GPUs can now reach FlyDSL kernels through existing operator APIs for dense and grouped GEMM, MXFP8/MXFP4 scaled GEMM, RMSNorm, TopK, and attention via AOTriton. Those operators previously ran on ATen, Triton, or AITER ASM implementations. On MI355X operator suites, FlyDSL gives a 1.10x geometric-mean speedup over the faster ATen/Triton baseline at each of 15 BF16 dense-GEMM shapes, and 1.58x and 1.68x over ATen for MXFP8 and MXFP4. End-to-end vLLM whole-request speedups reach 1.13x for BF16, 2.04x for MXFP8, and 3.01x for MXFP4, depending on model and concurrency.

PyTorch

FlyDSL Grouped GEMM MXFP scaled GEMM FMHA attention kernels