Reasoning-model RL training stops tuning only execution: RPEG also searches pipeline structure, up to 2x faster than ReaL on 256 GH200 GPUs
Synopsis
Reinforcement-learning training of reasoning language models chains generation, assessment and training across several large models each step, and existing systems usually optimize only part of that design space. RPEG places pipeline structure, including bounded asynchrony and actor–critic sharing, and execution choices such as stage fusion, fine-grained scheduling and multi-dimensional parallelism into one graph searched jointly, backed by a hardware-calibrated analytical performance model. On up to 256 GH200 GPUs its runtime is up to 2x faster end to end than ReaL, cuts step latency by up to 2.48x versus verl, and raises hardware utilization by up to 30%.
Interpretation
Reasoning-language-model RL post-training is optimized as a searchable graph in which pipeline structure itself is an optimization variable, not merely a fixed graph mapped onto hardware. Earlier systems each covered only part of the design space: ReaL and HybridFlow/verl chose placement and parallel strategies, RLHFuse, AReaL, Laminar and StaleFlow pursued fine-grained or asynchronous execution, and other work adapted resource orchestration. On up to 256 GH200 GPUs, the distributed runtime implementing RPEG improves end-to-end performance by up to 2x over ReaL and, in a controlled synchronous comparison with separate actor and critic models, achieves up to 2.477x lower time per step than verl (median 1.488).
Each candidate configuration is unrolled into a finite-step acyclic graph, then lowered under its execution plan into an executable graph with explicit communication and resharding nodes, timed by the analytical model, ordered by a greedy scheduler and scored. The analytical performance model jointly accounts for compute, communication and memory plus hierarchical rack-, node- and GPU-level communication costs, and uses automated hardware calibration instead of nominal vendor peaks. Across 123 configurations spanning 3B, 7B and 13B models, 1 to 128 GPUs and 25 parallel strategies, execution-time prediction reaches 15.5% mean absolute percentage error and peak-memory prediction 15.1%.
The runtime dispatches the selected plan on completion events and verifies that executed dependencies, parameter versions, transfers and resource ownership match the plan the optimizer chose. The optimizer searches structure and execution together, so any runtime deviation from the selected configuration would break both the predicted-to-measured correspondence and training semantics. On GH200, across 20 measured comparisons from seven Llama 3.1 8B baselines, every optimized configuration beats its own baseline by 1.250 to 1.471x, cutting graph execution time by roughly 20% to 32%.
Perspective
Measurements cover A100 and GH200 clusters, with end-to-end comparisons up to 256 GH200 GPUs; analytical-model validation covers 3B, 7B and 13B models, 1 to 128 GPUs and 25 parallel strategies. The authors propose that graph-native orchestration could extend to multi-agent, tool-augmented training and inference-time reasoning, but report no measurements for those settings.
The analytical model systematically underpredicts for inference and generation, which the authors attribute to roughly 2–3 GB per GPU of unmodeled CUDA context overhead absorbed by the reserved margin. Ablations show mechanism effects depend on workload: fixing microbatch counts improves the two smaller configurations by 2.1% to 2.4% but reduces the 70B case by 17.5%. Evidence of comparable training outcomes comes from mean rollout reward against synchronous verl, within 0.47% relative difference.