Software agents no longer get only pass or fail: the policy itself ranks debugging traces within the same outcome
Synopsis
Reinforcement learning rewards for software engineering agents no longer stop at pass or fail: the current policy compares debugging traces from the same task and verifier outcome, turning those preferences into bounded auxiliary rewards and lifting Qwen3-14B average accuracy on SWE-bench Verified from 32.1% to 34.7%.
Interpretation
Training rewards for software engineering agents now distinguish debugging quality inside the same verifier outcome, instead of only pass or fail. Previously a verifiable reward gave pass or fail alone, so a failed trace that located relevant code and gathered useful evidence scored the same zero as one that made no progress. On SWE-bench Verified, Qwen3-14B under a 400-step training budget rises from 32.1% to 34.7% average accuracy, and on SWE-bench-Live the same setting rises from 6.33% to 8.11%.
The method reuses the current policy as a post-rollout judge that compares only trajectories sharing a task and verifier outcome. Previously the extra dense signal came from a separately trained reward model, whereas here the policy itself supplies the comparisons without training a separate trajectory scorer. The judge scores observable debugging behavior, aggregates pairwise preferences into task-local scores, then normalizes, applies a soft deadzone, centers and bounds them, guaranteeing every passing trajectory outranks every failing one.
Refining only the failure bucket gives the strongest result. Extending refinement to successful trajectories adds judging workload without beating failure-only refinement on either reported metric. Failure-only refinement reaches 33.2% average accuracy and 46.6% Pass@3, refining both buckets gives 33.1% and 44.6%, and success-only refinement gives 32.0% and 44.0%.
Perspective
The method is validated on repository-level software engineering tasks with executable verifiers, trained on a filtered R2E-Gym subset and evaluated on SWE-bench Verified, SWE-bench-Live Lite and Terminal-Bench 2.1 with Qwen3-14B and Qwen3-32B. It suits researchers training software engineering agents who already have executable tests and multiple samples per task.
Agreement between the self-judge and the external judge Kimi K2.5 is roughly 0.6 to 0.7, declining somewhat at later checkpoints while downstream performance keeps improving. Training wall-clock time is about 19.86% higher than verifier-only RL, and a fully compute-matched comparison has not been run.