TorchTitan folds reinforcement learning into one model copy, so training and vLLM inference stop drifting apart

TorchTitan brings reinforcement learning into the same PyTorch-native training stack, keeping one model across pretraining, post-training and inference inside vLLM so users can work in a single codebase. Many reinforcement learning stacks keep separate training and inference implementations of the model, so architecture changes must stay aligned across both, and a run that goes sideways is hard to attribute to the reward, the algorithm, or the two copies disagreeing. TitanRL was tested on several workloads covering long context, code execution, compaction, harness and self-play, and is reported to train stably on MoE and dense models for hundreds of steps.

TitanRL offers a batch-invariant mode that drives the numerical gap between trainer and generator to zero in supported on-policy configurations. When rewards differ from a baseline after enabling a new algorithm or kernel, it was previously hard to tell whether the change or noise caused the difference. In supported on-policy configurations the trainer and generator produce bitwise identical log probabilities, so the gap between them is exactly zero at every step; a recent study with the TorchTitan team uses this to examine numerical mismatch in math and terminal-agent reinforcement learning.

TorchTitan added PyTorch-native support for Muse Spark Open, Muse Glimmer, Qwen3-VL and Qwen3.5, Kimi K2.7, Kimi K3 and DeepSeek-V4 across its 5D parallelism stack. These integrations were not previously present, and changing parallelization strategies, model components or hardware-specific kernels often meant maintaining separate model forks. The work introduces a composable configuration system built on typed Python recipes, model-config transforms and targeted component overrides, and integrates Attention Gym kernels including Gated DeltaNet, Kimi Delta Attention and sparse attention, plus opt-in Helion kernels such as fused RoPE, while remaining compatible with torch.compile.

FlexShard and torch_checkpointing give new implementation paths for optimizer updates, parameter communication and checkpoint writes in distributed training. FSDP2 sharding policies cannot express the layouts that the distributed Muon optimizer and blockwise parameter quantization need, and DCPv1 saves carry synchronization overhead that grows with scale. On DeepSeekV3 at 256 H100s, Muon parameters sharded with Owned and BlockShard placements improve optimizer time by up to 79% over an equivalent FSDP-only implementation; on LLama3 405B at 256 H100s, FP8 parameter AllGather communication volume drops by about 50% and end-to-end training step time improves by up to 20%; on DSv3 671B at 512 H100s with checkpoint interval 50, torch_checkpointing cuts average trainer blocking time from 841.2 ms to 34.1 ms.

The update addresses engineers and researchers doing large-model pretraining and post-training with PyTorch, and its reported scope is TorchTitan's measurements on the team's internal GB300 and H100 clusters plus TitanRL's training behavior on the listed workloads.

Bitwise agreement in batch-invariant mode holds only in supported on-policy configurations; the DeepSeek V3 671B throughput and numerics come from the team's internal GB300 cluster, scale only the PP, DP and EP axes, and use the MXFP8 quantization scheme.