# Agentic RL

RadixArk, working with NVIDIA, ran every sandbox of its open-source RL post-training framework Miles on NVIDIA Vera CPUs, which it calls the first public result of end-to-end agentic RL training with all sandboxes on Vera CPU servers. In a comparison that first kept the machines busy, with the same model, tasks, GPUs and settings, episodes with sandboxes on Turin took 12% longer on average, while model thinking time was identical, so nearly all of the gap came from work inside the sandbox. With a single sandbox running, Vera performs on par with or better than current x86, and its lead grows with load.

RadixArk, working with NVIDIA, ran every sandbox of its open-source RL post-training framework Miles on NVIDIA Vera CPUs, which it calls the first public result of end-to-end agentic RL training with all sandboxes on Vera CPU servers. In a comparison that first kept the machines busy, with the same model, tasks, GPUs and settings, episodes with sandboxes on Turin took 12% longer on average, while model thinking time was identical, so nearly all of the gap came from work inside the sandbox. With a single sandbox running, Vera performs on par with or better than current x86, and its lead grows with load.

Agentic reinforcement learning shifts the reasoning trajectories of large language models, and trajectory-based confidence estimators trained on a base model lose much of their ranking ability after that shift. StaR-TTA treats trajectory stability as a label-free reliability signal: it learns the link between stability and correctness on labeled base trajectories, then uses stability rankings from unlabeled RL trajectories to correct confidence at test time. Across four QA benchmarks it reaches an average AUROC of 0.7208, with gains on unseen datasets, different model families and scales, and both PPO and GRPO.

Agentic reinforcement learning shifts the reasoning trajectories of large language models, and trajectory-based confidence estimators trained on a base model lose much of their ranking ability after that shift. StaR-TTA treats trajectory stability as a label-free reliability signal: it learns the link between stability and correctness on labeled base trajectories, then uses stability rankings from unlabeled RL trajectories to correct confidence at test time. Across four QA benchmarks it reaches an average AUROC of 0.7208, with gains on unseen datasets, different model families and scales, and both PPO and GRPO.

Tool-using language-model agents often need dozens of interactions before a single success-or-failure label arrives, leaving no way to tell which step went wrong. STEP-VERIFIER turns a success-labeled corpus of logged interactions into action-level comparisons: the difference in average final return between two continuations that share a common prefix labels which of two actions was better, and a Bradley-Terry ranking head converts those pairs into calibrated per-step scores. On Llama-3.1-8B-Instruct and Mistral-7B-Instruct backbones, it beats outcome-only GRPO on the average metric in all six benchmark-backbone combinations across GAIA, ToolBench and AgentBoard, and its per-action scores agree with human judgments on 81.5% of pairs.

Tool-using language-model agents often need dozens of interactions before a single success-or-failure label arrives, leaving no way to tell which step went wrong. STEP-VERIFIER turns a success-labeled corpus of logged interactions into action-level comparisons: the difference in average final return between two continuations that share a common prefix labels which of two actions was better, and a Bradley-Terry ranking head converts those pairs into calibrated per-step scores. On Llama-3.1-8B-Instruct and Mistral-7B-Instruct backbones, it beats outcome-only GRPO on the average metric in all six benchmark-backbone combinations across GAIA, ToolBench and AgentBoard, and its per-action scores agree with human judgments on 81.5% of pairs.

On AMD Instinct MI355X, Qwen3-30B-A3B-Instruct trained with fully asynchronous agentic RL after supervised fine-tuning raised pass@1 from 7.3% to 27.5% on a 100-task stratified subset of SWE-bench Verified, with RL tasks drawn from repositories absent from that benchmark. The bottlenecks were long-tail trajectories and CPU sandbox scheduling rather than GPU compute: capping trajectory length cut step time from 850 s to 284 s, and cloud sandboxes at 384 concurrent trajectories lifted rollout throughput from about 860 to about 3,480 trajectories per hour.

On AMD Instinct MI355X, Qwen3-30B-A3B-Instruct trained with fully asynchronous agentic RL after supervised fine-tuning raised pass@1 from 7.3% to 27.5% on a 100-task stratified subset of SWE-bench Verified, with RL tasks drawn from repositories absent from that benchmark. The bottlenecks were long-tail trajectories and CPU sandbox scheduling rather than GPU compute: capping trajectory length cut step time from 850 s to 284 s, and cloud sandboxes at 384 concurrent trajectories lifted rollout throughput from about 860 to about 3,480 trajectories per hour.

Step 5 Preview treats agentic workloads as the starting point for model design: it uses a narrow-but-deep stack of 92 transformer blocks, interleaves efficient and sparse full attention at 3:1, and extends the full-attention path to contexts up to 1M tokens with Sparse GQA. Its total parameter count is under one-quarter and its active count under one-third of trillion-scale models such as Qwen3.8-Max and Kimi K3, and it reports long-horizon agentic capability comparable to those counterparts. Agent-native data synthesis and long-rollout RL infrastructure accompany the model, yielding million-level verifiable environments.

Step 5 Preview treats agentic workloads as the starting point for model design: it uses a narrow-but-deep stack of 92 transformer blocks, interleaves efficient and sparse full attention at 3:1, and extends the full-attention path to contexts up to 1M tokens with Sparse GQA. Its total parameter count is under one-quarter and its active count under one-third of trillion-scale models such as Qwen3.8-Max and Kimi K3, and it reports long-horizon agentic capability comparable to those counterparts. Agent-native data synthesis and long-rollout RL infrastructure accompany the model, yielding million-level verifiable environments.