Trillion-Parameter Pretraining Goes Bitwise-Deterministic: NVIDIA Cuts the Determinism Tax to About 2%

Trillion-parameter pretraining can now run close to nondeterministic speed while staying bitwise deterministic, so failure replay, loss-spike isolation and checkpoint resume all rest on a reproducible numerical trajectory. Deterministic recipes previously carried double-digit percentage overhead, enough to make production training impractical, and printed loss values are not sensitive enough because two runs can print the same rounded loss while differing in the lower-order bits of gradients and parameters. Optimization across three workloads reduced step-time determinism overhead from double-digit baselines to low single digits: Nemotron 3 Ultra from about 17% to 2%, then to 1.5% at 3,072 GPUs; the hybrid Triton proxy from about 60% to 37%, then to 2%; the CuTeDSL weight-gradient path measured about 3.6% at 256 GPUs.

Determinism comes from fixing reduction order and keeping a full-precision source of truth, not from comparing loss curves after the fact. In the grouped-GEMM epilogue, multiple N-tiles originally accumulated into the same dprob[token] address, making the result depend on their arrival order; serializing those writers restored determinism but reduced parallelism. Each N-tile gets a private output slot, parallel execution inside the kernel is preserved, and slots are combined in a fixed order after all writers finish; the checkpoint path preserves a full-precision source of truth and reconstructs the runtime representation through the same conversion path used in training. The large-scale Nemotron recipe measured about 2% steady-state overhead at 2,432 GPUs with bitwise determinism over 800 steps.

The approach targets teams pretraining trillion-parameter models on thousands of GPUs who need reproducible failure replay and checkpoint resume, and validation is done within the same hardware and software environment. The Megatron Core User Guide points to a supported recipe, after which independent and checkpoint-resumed runs can be compared to validate reproducibility.

Bitwise determinism is validated only within the same hardware and software environment; comparisons across GPU generations, network configurations or library versions are outside this validation scope and may produce different results. A low determinism tax on a few GPUs does not guarantee the same result at production scale, since communication, synchronization, pipeline bubbles, expert routing and load balance can change the relative overhead, and new kernels, fusions, precision formats or parallelism configurations can reintroduce nondeterminism.