# Elastic determinism

Changing the GPU count during training usually changes the order in which a data loader feeds samples to the model, so an ablation can end up measuring hardware allocation instead of a data change. Zephon splits each loading operator into a single-threaded accumulator that makes stateful decisions and stateless parallel workers, and divides data into a fixed number of logical lanes; plugged into TorchTitan in place of the default loader, it resumed a 16-GPU checkpoint on 8, 32 and 64 GPUs with every global batch matching the original run's SHA-256 hash.

Changing the GPU count during training usually changes the order in which a data loader feeds samples to the model, so an ablation can end up measuring hardware allocation instead of a data change. Zephon splits each loading operator into a single-threaded accumulator that makes stateful decisions and stateless parallel workers, and divides data into a fixed number of logical lanes; plugged into TorchTitan in place of the default loader, it resumed a 16-GPU checkpoint on 8, 32 and 64 GPUs with every global batch matching the original run's SHA-256 hash.

Zephon is a data loader for foundation models that provides elastic determinism for online, stateful pipelines (such as online tokenization, packing, and mixing of samples), meaning a deterministic global training batch sequence despite changes in GPU topology, frequent checkpoint-resume cycles, and different data processing backends, while checkpointing only bounded in-flight state so recovery cost does not grow with training progress; it achieves competitive throughput on text and vision-language workloads and offers a combination of guarantees no existing loader provides for online, stateful pipelines.

Zephon is a data loader for foundation models that provides elastic determinism for online, stateful pipelines (such as online tokenization, packing, and mixing of samples), meaning a deterministic global training batch sequence despite changes in GPU topology, frequent checkpoint-resume cycles, and different data processing backends, while checkpointing only bounded in-flight state so recovery cost does not grow with training progress; it achieves competitive throughput on text and vision-language workloads and offers a combination of guarantees no existing loader provides for online, stateful pipelines.