# SWE-bench Verified

RadixArk, working with NVIDIA, ran every sandbox of its open-source RL post-training framework Miles on NVIDIA Vera CPUs, which it calls the first public result of end-to-end agentic RL training with all sandboxes on Vera CPU servers. In a comparison that first kept the machines busy, with the same model, tasks, GPUs and settings, episodes with sandboxes on Turin took 12% longer on average, while model thinking time was identical, so nearly all of the gap came from work inside the sandbox. With a single sandbox running, Vera performs on par with or better than current x86, and its lead grows with load.

RadixArk, working with NVIDIA, ran every sandbox of its open-source RL post-training framework Miles on NVIDIA Vera CPUs, which it calls the first public result of end-to-end agentic RL training with all sandboxes on Vera CPU servers. In a comparison that first kept the machines busy, with the same model, tasks, GPUs and settings, episodes with sandboxes on Turin took 12% longer on average, while model thinking time was identical, so nearly all of the gap came from work inside the sandbox. With a single sandbox running, Vera performs on par with or better than current x86, and its lead grows with load.

MemUse moves the unit of evaluation from the task outcome to the individual memory delivery: across 8,000 SWE-bench Verified agent episodes, three memory systems stayed within 0.4 percentage points of a no-memory baseline on resolve rate, yet per-delivery judgments show most delivered entries were irrelevant, 51.3%–79.7% of relevant entries went unused in each system, and 23.2% of episodes with beneficial memory use still failed.

MemUse moves the unit of evaluation from the task outcome to the individual memory delivery: across 8,000 SWE-bench Verified agent episodes, three memory systems stayed within 0.4 percentage points of a no-memory baseline on resolve rate, yet per-delivery judgments show most delivered entries were irrelevant, 51.3%–79.7% of relevant entries went unused in each system, and 23.2% of episodes with beneficial memory use still failed.

On AMD Instinct MI355X, Qwen3-30B-A3B-Instruct trained with fully asynchronous agentic RL after supervised fine-tuning raised pass@1 from 7.3% to 27.5% on a 100-task stratified subset of SWE-bench Verified, with RL tasks drawn from repositories absent from that benchmark. The bottlenecks were long-tail trajectories and CPU sandbox scheduling rather than GPU compute: capping trajectory length cut step time from 850 s to 284 s, and cloud sandboxes at 384 concurrent trajectories lifted rollout throughput from about 860 to about 3,480 trajectories per hour.

On AMD Instinct MI355X, Qwen3-30B-A3B-Instruct trained with fully asynchronous agentic RL after supervised fine-tuning raised pass@1 from 7.3% to 27.5% on a 100-task stratified subset of SWE-bench Verified, with RL tasks drawn from repositories absent from that benchmark. The bottlenecks were long-tail trajectories and CPU sandbox scheduling rather than GPU compute: capping trajectory length cut step time from 850 s to 284 s, and cloud sandboxes at 384 concurrent trajectories lifted rollout throughput from about 860 to about 3,480 trajectories per hour.

Reinforcement learning rewards for software engineering agents no longer stop at pass or fail: the current policy compares debugging traces from the same task and verifier outcome, turning those preferences into bounded auxiliary rewards and lifting Qwen3-14B average accuracy on SWE-bench Verified from 32.1% to 34.7%.

Reinforcement learning rewards for software engineering agents no longer stop at pass or fail: the current policy compares debugging traces from the same task and verifier outcome, turning those preferences into bounded auxiliary rewards and lifting Qwen3-14B average accuracy on SWE-bench Verified from 32.1% to 34.7%.

ARIA turns when to consult a stronger model into a learned sequential decision: across SWE-bench Verified, Terminal-Bench 2.1 and a telecom customer-service benchmark it recovers 44–81% of the executor–advisor success gap at less than half the Advisor-only cost.

ARIA turns when to consult a stronger model into a learned sequential decision: across SWE-bench Verified, Terminal-Bench 2.1 and a telecom customer-service benchmark it recovers 44–81% of the executor–advisor success gap at less than half the Advisor-only cost.

Supervised fine-tuning of coding agents no longer has to select only on whether the final patch passes the test suite: ranking trajectories by structure and keeping the top 1,024 raised pass@1 on SWE-Bench Verified from 24.8% for a random subset to 27.6%.

Supervised fine-tuning of coding agents no longer has to select only on whether the final patch passes the test suite: ranking trajectories by structure and keeping the top 1,024 raised pass@1 on SWE-Bench Verified from 24.8% for a random subset to 27.6%.

Repository-level software engineering agents previously treated each attempt's pass or fail as a terminal reward, discarding the failed patch and verifier feedback. FC-SWE restores the repository to its original task state after a patch fails verification, uses the failed patch and verifier diagnostics as context for a recovery trajectory, keeps each trajectory's own verifier outcome, and compares only the initial and recovery trajectories actually executed for the same issue. On all 500 SWE-bench Verified tasks with the same Qwen3.5-4B policy, first-attempt resolution rises from GRPO's 38.9% to 41.7% and two-attempt cumulative resolution from 48.5% to 52.8%; trained with at most two attempts per chain, the same policy reaches 70.7% with eleven test-time attempts.

Repository-level software engineering agents previously treated each attempt's pass or fail as a terminal reward, discarding the failed patch and verifier feedback. FC-SWE restores the repository to its original task state after a patch fails verification, uses the failed patch and verifier diagnostics as context for a recovery trajectory, keeps each trajectory's own verifier outcome, and compares only the initial and recovery trajectories actually executed for the same issue. On all 500 SWE-bench Verified tasks with the same Qwen3.5-4B policy, first-attempt resolution rises from GRPO's 38.9% to 41.7% and two-attempt cumulative resolution from 48.5% to 52.8%; trained with at most two attempts per chain, the same policy reaches 70.7% with eleven test-time attempts.

DeReAct introduces a modular agent architecture that separates action authorization and completion control from a single LLM policy, delegating them to a Critic that validates proposed actions before execution and a Context Manager that reconstructs an environment-supported State and certifies task completion; across GAIA and SWE-bench Verified it improves Pass@1 most for weaker Brain models (6.5–7.0 points for Qwen3-Coder-480B and 4.2–5.2 points for Claude Sonnet 4.5), with gains diminishing as Brain capability increases, while on Claude Opus 4.5 Pass@1 remains comparable to ReAct but trajectories are more evidence-complete and constraint-satisfying.

DeReAct introduces a modular agent architecture that separates action authorization and completion control from a single LLM policy, delegating them to a Critic that validates proposed actions before execution and a Context Manager that reconstructs an environment-supported State and certifies task completion; across GAIA and SWE-bench Verified it improves Pass@1 most for weaker Brain models (6.5–7.0 points for Qwen3-Coder-480B and 4.2–5.2 points for Claude Sonnet 4.5), with gains diminishing as Brain capability increases, while on Claude Opus 4.5 Pass@1 remains comparable to ReAct but trajectories are more evidence-complete and constraint-satisfying.