Agents no longer learn from a scalar reward alone: ACE-RL trains the reflections and notes they hand forward, topping held-out reward on four benchmarks

Agent training no longer optimizes only the actions of the current attempt; the experience text the model writes for later attempts becomes a training target too. The two prior routes each lacked half: scalar-reward RL compresses a whole trial into one number and discards error logs, tool outputs and discovered constraints, while a fixed harness such as ACE passes reflections and notes to the next attempt but never trains the calls that write them. At equal training compute on Mastermind, MBPP, AppWorld and MiniSWEBench, ACE-RL has the highest held-out reward of the four, above actor-only GRPO, ERL, and delayed-joint PPO trained end to end through the harness with a learned critic.

The method fixes the ACE harness and has one model serve as actor, reflector and playbook curator, scoring each by what its candidate note does to a fresh attempt. A candidate playbook is handed to a fresh actor rollout and read by the task verifier, so no learned reward model is needed; each role is compared only with other outputs of the same input, reflections with other reflections of one trajectory and edits with other edits of one reflection. Training uses Qwen3-0.6B on Mastermind and MBPP and Qwen3-4B-Instruct-2507 on AppWorld and MiniSWEBench, with eight actor rollouts per task, four of them seed rollouts and the rest scoring candidate playbooks.

The gain comes from the playbook itself, not only from a better-trained actor. Discarding every edit so each visit reads the empty playbook lowers the same checkpoint's score. Within the training horizon, applying edits raises RSI@K at K=2 and K=3 on all four benchmarks, by 0.105±0.041 on AppWorld at K=2 and 0.090±0.040 on MiniSWEBench at K=3.

The results cover four benchmarks, Mastermind, MBPP, AppWorld and MiniSWEBench, models up to 4B parameters, one training run per method, and comparisons at equal training compute. Playbooks are task-local and every attempt starts a fresh environment, so transfer of experience across tasks is outside the studied scope.

Delayed-joint PPO is below ACE-RL on all four benchmarks and its test reward peaks early and then declines, yet its critic's explained variance stays above 0.95 late in each run, so a poorly fit critic does not explain the decline; the authors state they cannot conclude that critic-based credit cannot train the handoff. The edits-discarded control on AppWorld is evaluated only beyond its compute budget, the MBPP environment's feedback shows the scored assertions, and whether the same recipe is needed or sufficient at frontier scale remains open.