# Transformer Engine

NVIDIA wired Transformer Engine's MoE optimizations into the JAX training stack so every token reaches its selected expert, with no dropping and no padding. An unoptimized baseline training DeepSeek-V3 on GB300 reached only 103 TFLOPS/GPU, with inter-GPU communication consuming 84% of accumulated kernel time; grouped GEMM, MXFP8 grouped quantization and fused expert-parallel dispatch and combine raised that to 1,068 TFLOPS/GPU, about a tenfold end-to-end throughput gain, and the stack sustained 97% efficiency at 1,024 GPUs.

NVIDIA wired Transformer Engine's MoE optimizations into the JAX training stack so every token reaches its selected expert, with no dropping and no padding. An unoptimized baseline training DeepSeek-V3 on GB300 reached only 103 TFLOPS/GPU, with inter-GPU communication consuming 84% of accumulated kernel time; grouped GEMM, MXFP8 grouped quantization and fused expert-parallel dispatch and combine raised that to 1,068 TFLOPS/GPU, about a tenfold end-to-end throughput gain, and the stack sustained 97% efficiency at 1,024 GPUs.