Tao Ren

Long-horizon LLM agents interact with their environment through complete turns but generate each action token by token, and existing reinforcement learning either gives a whole trajectory one shared advantage or treats every token as an action. MPO splits credit assignment into two scales: turn-level generalized advantage estimation carries credit across interactions, a mean-preserving local correction refines it within each turn, and the policy update uses complete-turn importance ratios. Across search, tool-use and automation tasks, MPO reports the highest score on all nine metrics at both Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct scales, including a 4.33-point BrowseComp-Plus pass@1 gain over the strongest non-MPO baseline at 4B.

Long-horizon LLM agents interact with their environment through complete turns but generate each action token by token, and existing reinforcement learning either gives a whole trajectory one shared advantage or treats every token as an action. MPO splits credit assignment into two scales: turn-level generalized advantage estimation carries credit across interactions, a mean-preserving local correction refines it within each turn, and the policy update uses complete-turn importance ratios. Across search, tool-use and automation tasks, MPO reports the highest score on all nine metrics at both Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct scales, including a 4.33-point BrowseComp-Plus pass@1 gain over the strongest non-MPO baseline at 4B.