Multi-turn RL pushes LLM agents past the base model only with roomy budgets, and the edge fades or reverses when turns tighten

The improvement from multi-turn agentic reinforcement learning depends on how many interaction turns each trajectory is allowed, not only on whether the task is eventually completed. Agent evaluations usually report trajectory-level success, meaning whether a task finishes within a given budget, so they cannot separate doing better in fewer turns from recovering over more turns. With sampling breadth fixed and the maximum turn count swept from tight to loose, the RL model's advantage over the base model grows with the budget; under tight budgets it shrinks and reverses on some tasks, even while substantial headroom remains below each task's structural performance upper bound.

Multi-turn RL does not make every step better; it mainly strengthens recovery after errors and execution in later parts of a trajectory. If the gain came from uniformly stronger single-step decisions, the RL model should reliably produce higher-quality actions from the same state, and matched-state comparisons do not support that. Sampling actions from both policies at identical states and averaging their quality, the RL model is better on only 46.8% of DeepCoder-MT states and 42.8% of FrozenLake states; at error states built by injecting an incorrect action, the share reverses to 65.7% on FrozenLake and 67.3% on DeepCoder-MT.

Individual RL actions stay within the probability support of the base policy, yet complete trajectories are far more likely under the RL policy. Step-level action quality does not generally improve while trajectory success rises, so the gap between local actions and whole trajectories needs an explanation. On Terminal-Bench, RL actions have mean negative log-likelihood under the base policy close to that of base actions, while the cumulative log-likelihood ratio of successful RL trajectories rises with interaction progress to a level clearly above successful base trajectories.

Varying the interaction budget during training preserves long-horizon gains while easing the decline in single-step decision quality. Standard multi-turn RL samples trajectories under a fixed maximum budget and does not distinguish solutions that need different amounts of interaction, and during training trajectory reward rises while step reward does not improve and even declines. BudgetMix is compared with full-budget training under GRPO, GiGPO and PPO, averaging 7.5, 5.3 and 4.0 percentage points above full-budget RL at 25%, 50% and 75% budgets respectively, and matching it at the 100% budget.

The analysis covers interactive tasks including DeepCoder-MT, BFCL, InterCode-SQL, FrozenLake and Terminal-Bench, with main experiments on Qwen3-4B and Qwen3-8B using GRPO as the primary algorithm and appendix results for GiGPO, PPO, InternVL3.5-8B and DeepSeek-R1-Distill-Llama-8B; step-level action quality is measured only on FrozenLake and DeepCoder-MT, the two tasks with reliable step-level evaluation.

Error states are built by injecting an incorrect action, which the authors describe as a proxy rather than a rigorous definition; precise process supervision is available only in settings such as FrozenLake where Q-values can be computed, and on more realistic tasks such as BFCL dense-signal methods did not ease the drop under tight budgets; BudgetMix is presented as a candidate in a preliminary discussion.