Skip to main content

# Deterministic Execution

2 related articles

Subscribe to updates

NVIDIA Technical Blog Oct 7, 2026
Trillion-Parameter Pretraining Goes Bitwise-Deterministic: NVIDIA Cuts the Determinism Tax to About 2%
NVIDIA has pushed the cost of bitwise-deterministic pretraining in Megatron Core down to a workable range: two independent runs from the same initial state, and a run resumed from checkpoint, must stay numerically identical at every step. Such deterministic recipes previously carried double-digit overhead; after buffer-fill optimization Nemotron 3 Ultra fell from about 17% to 2%, then to 1.5% at 3,072 GPUs, and the large-scale Nemotron recipe measured about 2% steady-state overhead at 2,432 GPUs with bitwise determinism over 800 steps.
NVIDIA has pushed the cost of bitwise-deterministic pretraining in Megatron Core down to a workable range: two independent runs from the same initial state, and a run resumed from checkpoint, must stay numerically identical at every step. Such deterministic recipes previously carried double-digit overhead; after buffer-fill optimization Nemotron 3 Ultra fell from about 17% to 2%, then to 1.5% at 3,072 GPUs, and the large-scale Nemotron recipe measured about 2% steady-state overhead at 2,432 GPUs with bitwise determinism over 800 steps.
NVIDIA
Deterministic Execution Megatron Core Nemotron Reproducibility Mixture of Experts
NVIDIA Technical Blog Sep 16, 2026
NVIDIA cuts voltage droop by over 60% with Groq 3 LPX deterministic execution to get more inference tokens per megawatt
NVIDIA describes the deterministic execution model of the Groq 3 LPX low-latency accelerator within its Vera Rubin platform: the LPU compiler schedules compute and data movement down to the clock cycle, letting it predict per-cycle current demand and use Preemptive Power (PEP) and Clock Period Synthesis (CPS) to reduce voltage droop and shrink the voltage guardband, with internal testing showing over 60% less voltage drop, an estimated high single-digit percentage decrease in the baseline voltage that must be continually supplied, a potentially low-double-digit percentage reduction in power for the same workload, and up to 35x higher throughput per megawatt at the platform level versus the previous-generation GB200 NVL72 for 2T+ parameter models at long context and high interactivity.
NVIDIA describes the deterministic execution model of the Groq 3 LPX low-latency accelerator within its Vera Rubin platform: the LPU compiler schedules compute and data movement down to the clock cycle, letting it predict per-cycle current demand and use Preemptive Power (PEP) and Clock Period Synthesis (CPS) to reduce voltage droop and shrink the voltage guardband, with internal testing showing over 60% less voltage drop, an estimated high single-digit percentage decrease in the baseline voltage that must be continually supplied, a potentially low-double-digit percentage reduction in power for the same workload, and up to 35x higher throughput per megawatt at the platform level versus the previous-generation GB200 NVL72 for 2T+ parameter models at long context and high interactivity.
NVIDIA
NVIDIA Groq 3 LPX Deterministic Execution NVIDIA Vera Rubin NVL72 NVIDIA DSX MaxLPS AI factory

You’re all caught up