Software-engineering agents stop waiting for the slowest rollout: async RL lifts pass@1 from 7.3% to 27.5%
Synopsis
On AMD Instinct MI355X, Qwen3-30B-A3B-Instruct trained with fully asynchronous agentic RL after supervised fine-tuning raised pass@1 from 7.3% to 27.5% on a 100-task stratified subset of SWE-bench Verified, with RL tasks drawn from repositories absent from that benchmark. The bottlenecks were long-tail trajectories and CPU sandbox scheduling rather than GPU compute: capping trajectory length cut step time from 850 s to 284 s, and cloud sandboxes at 384 concurrent trajectories lifted rollout throughput from about 860 to about 3,480 trajectories per hour.
Interpretation
Agentic RL for software-engineering agents no longer waits on its slowest trajectory; the trainer updates once enough completed trajectories are available instead of finishing a whole group. In the earlier synchronous flow, two attempts at the same task could differ 2.5x in length, and a GRPO group could not be scored until its slowest attempt finished, so a few stragglers held up the step. On 100 stratified SWE-bench Verified tasks with 4 attempts each, the model's pass@1 rose from 7.3% to 27.5%, with supervised fine-tuning first lifting it to 12.8% and agentic RL adding 14.7 points.
The stack separates inference from training: the inference node runs eight single-GPU SGLang engines plus Harbor for sandboxes, and the training node runs Megatron-LM and pushes new weights back to the engines. Beyond the long tail, every trajectory needs its own sandbox to build and test code, making sandboxes a CPU fleet on the critical path while agents wait and GPU slots idle. On MI355X with 288 GB HBM3E per GPU, moving sandboxes to E2B cloud microVMs left the engines with an empty queue and at most 23% KV-cache use at 384 concurrent trajectories, with cloud sandbox start at a 1.9 s median versus 11.7 s for local Docker under load.
Capping trajectory length and engine-side context removes the long tail and KV-cache thrashing from the training loop. Without a context cap, a few generations grew to 113K-186K tokens, and about 10 of them filled the engine's 1.5M-token KV cache, leaving the other 99 requests on that engine without tokens until they timed out. On the same 8-GPU inference node, a tighter length cap cut training step time from 850 s to 284 s, about 3x faster, with no effect on successful runs.
Perspective
The results come from training and evaluating Qwen3-30B-A3B-Instruct on AMD Instinct MI355X, evaluated on a 100-task stratified sample of SWE-bench Verified with 4 attempts per task; throughput and step time come from different hardware comparisons rather than one fixed-hardware run.
The authors report a 145-step run that survived three restarts from checkpoint with no drift or blow-up in the curves, and note that 3.1% of trajectories ended in an infrastructure error, which scores 0 just like a wrong fix and must be logged and filtered.