# TorchTitan

TorchTitan's annual update adds PyTorch-native training support for open-weight models including Muse Spark Open, Qwen3.5, Kimi K3 and DeepSeek-V4, and introduces TitanRL, which keeps one model from pretraining through post-training to inside vLLM so training and inference no longer run two copies that can quietly disagree. In supported on-policy configurations the trainer and generator produce bitwise identical log probabilities, leaving exactly zero gap at every step. The update also reports DeepSeek V3 671B throughput on the MLPerf training benchmark and infrastructure work around FlexShard and torch_checkpointing.

TorchTitan's annual update adds PyTorch-native training support for open-weight models including Muse Spark Open, Qwen3.5, Kimi K3 and DeepSeek-V4, and introduces TitanRL, which keeps one model from pretraining through post-training to inside vLLM so training and inference no longer run two copies that can quietly disagree. In supported on-policy configurations the trainer and generator produce bitwise identical log probabilities, leaving exactly zero gap at every step. The update also reports DeepSeek V3 671B throughput on the MLPerf training benchmark and infrastructure work around FlexShard and torch_checkpointing.

Changing the GPU count during training usually changes the order in which a data loader feeds samples to the model, so an ablation can end up measuring hardware allocation instead of a data change. Zephon splits each loading operator into a single-threaded accumulator that makes stateful decisions and stateless parallel workers, and divides data into a fixed number of logical lanes; plugged into TorchTitan in place of the default loader, it resumed a 16-GPU checkpoint on 8, 32 and 64 GPUs with every global batch matching the original run's SHA-256 hash.

Changing the GPU count during training usually changes the order in which a data loader feeds samples to the model, so an ablation can end up measuring hardware allocation instead of a data change. Zephon splits each loading operator into a single-threaded accumulator that makes stateful decisions and stateless parallel workers, and divides data into a fixed number of logical lanes; plugged into TorchTitan in place of the default loader, it resumed a 16-GPU checkpoint on 8, 32 and 64 GPUs with every global batch matching the original run's SHA-256 hash.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.