# Megatron Core

NVIDIA has pushed the cost of bitwise-deterministic pretraining in Megatron Core down to a workable range: two independent runs from the same initial state, and a run resumed from checkpoint, must stay numerically identical at every step. Such deterministic recipes previously carried double-digit overhead; after buffer-fill optimization Nemotron 3 Ultra fell from about 17% to 2%, then to 1.5% at 3,072 GPUs, and the large-scale Nemotron recipe measured about 2% steady-state overhead at 2,432 GPUs with bitwise determinism over 800 steps.

NVIDIA has pushed the cost of bitwise-deterministic pretraining in Megatron Core down to a workable range: two independent runs from the same initial state, and a run resumed from checkpoint, must stay numerically identical at every step. Such deterministic recipes previously carried double-digit overhead; after buffer-fill optimization Nemotron 3 Ultra fell from about 17% to 2%, then to 1.5% at 3,072 GPUs, and the large-scale Nemotron recipe measured about 2% steady-state overhead at 2,432 GPUs with bitwise determinism over 800 steps.