Long-horizon agents should bank credit per turn: MPO splits turn-level advantage from token refinement, lifting search pass@1 by up to 4.33 points
Synopsis
Long-horizon LLM agents interact with their environment through complete turns but generate each action token by token, and existing reinforcement learning either gives a whole trajectory one shared advantage or treats every token as an action. MPO splits credit assignment into two scales: turn-level generalized advantage estimation carries credit across interactions, a mean-preserving local correction refines it within each turn, and the policy update uses complete-turn importance ratios. Across search, tool-use and automation tasks, MPO reports the highest score on all nine metrics at both Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct scales, including a 4.33-point BrowseComp-Plus pass@1 gain over the strongest non-MPO baseline at 4B.
Interpretation
Policy optimization for long-horizon agents can now be organized at both the turn and the token scale: task credit propagates across turns, and tokens within a turn are distinguished again. Critic-free, group-relative methods such as GRPO, DAPO and GSPO derive one shared advantage per generated sequence from group-relative outcome rewards and do not distinguish individual turns or tokens; critic-based token-level methods such as PPO, VAPO and SAO treat each generated token as an action, giving finer credit without explicitly separating within-turn generation from turn-level environment interaction. Across 2,048 trajectories and 16,323 turns from eight checkpoint batches of one search-agent run, differences between turn means account for a median 89.5% of the critic's token-weighted within-trajectory value variance, and between-turn variance exceeds within-turn variance in 93.8% of trajectories.
MPO uses Multiscale Advantage Estimation (MAE) to propagate credit along the turn axis first, then allocates it among the tokens of each turn. The turn-level part of MAE runs generalized advantage estimation on turn boundaries, so shared credit decays once per interaction rather than once per generated token; the within-turn part uses a local trace restricted to the current turn and a centered least-squares projection that keeps the turn mean fixed. The policy update applies complete-turn importance ratios and uses a stop-gradient construction so each token's MAE advantage stays in its own score gradient; comparisons cover GRPO, GSPO, PPO, VAPO and SAO on search, Claw-Eval and Automation Bench at both the 4B and 30B-A3B model scales.
Replacing turn-level importance ratios with token-level ratios lowers every reported score, so the two alignments each contribute. A variant that keeps MAE but uses token-level importance ratios drops BrowseComp-Plus, GAIA and HLE pass@1 by 2.36, 3.12 and 1.03 points at 4B, and by 2.05, 2.19 and 0.58 points at 30B-A3B. That variant still beats PPO on all six search metrics, indicating that multiscale credit assignment contributes gains on its own, with complete-turn importance correction adding further improvement.
Perspective
The demonstrated scope is separate training of Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct on search (BrowseComp-Plus, GAIA, HLE), Claw-Eval and Automation Bench, with 300 training steps for search, 300 for Claw-Eval and 200 for Automation Bench; each run selects its best-performing checkpoint and reports the mean of five offline evaluations.
Critic explained variance is higher under MPO, but it is measured on each policy's own rollouts and also depends on return variance, so it does not isolate a causal improvement in critic accuracy; each ablation setting has one training run, and the Claw and Automation training data are proprietary with details undisclosed.