Agents no longer learn from a scalar reward alone: ACE-RL trains the reflections and notes they hand forward, topping held-out reward on four benchmarks
Agent reinforcement learning usually compresses a trial into one scalar reward, discarding the error logs and tool outputs the environment revealed; ACE-RL also trains the reflections and task notes the model writes, scoring each candidate note by a fresh attempt that executes it. At equal training compute on Mastermind, MBPP, AppWorld and MiniSWEBench, ACE-RL's held-out reward is higher than actor-only GRPO, ERL, and delayed-joint PPO trained end to end through the harness with a learned critic.