Tool calls get worse with training: TACA supervises the reasoning-to-call transition and lifts six-benchmark accuracy by 4.61 points
Synopsis
In multi-turn tool-use reinforcement learning, reasoning accuracy can rise early and then fall below the non-agentic baseline; TACA traces this to negative aggregate credit on tool-call tokens and instead supervises the step where reasoning turns into a tool call. Against MT-GRPO it raises mean accuracy over six reasoning benchmarks by 4.61 and 3.61 percentage points for Qwen2.5-1.5B and 3B, while tool execution success during training stays at 86.28%.
Interpretation
In multi-turn tool-use reinforcement learning, tool-calling ability and reasoning scores can decline together during training, and the problem lies in the credit assigned to tool-call tokens. MT-GRPO spreads a trajectory's terminal reward across every generated turn, so tool-call tokens and reasoning tokens share one advantage value, leaving training degradation without an account tied to tool interaction. Training Qwen2.5-3B-Instruct on DAPO-Math, validation accuracy improves early and then falls below the 35% non-agentic reference, reaching 26.64% at step 500, while tool-use rate stays broadly stable and execution success drops from 69.41% to 48.41%.
TACA aims supervision at the switch from reasoning to tool use, adjusting call-token advantages and reusing previously sampled calls. MT-GRPO assigns credit only from the final answer, so a successful call can be penalized when later reasoning fails; TACA closes this gap with Advantage Projection, Decision Contrast and Supervised Replay. Advantage Projection applies a minimal uniform shift to tool-call tokens within each prompt group so their mean advantage is nonnegative; Decision Contrast compares rewards of tool-using and direct-answer trajectories for the same prompt; Supervised Replay stores calls from correct trajectories with their original context and draws up to 32 pairs per update for likelihood training.
Across four models, TACA's mean accuracy exceeds MT-GRPO's, and tool use does not disappear during training. MT-GRPO reaches six-benchmark mean accuracies of 28.63%, 39.90%, 42.59% and 13.61% on Qwen2.5-1.5B, 3B, 7B and Llama-3.2-1B. TACA reaches 33.24%, 43.51%, 45.97% and 15.17% on the same models; on Qwen2.5-1.5B the TACO unit-test average rises from 41.98 to 53.70 and WebShop success rate from 56.8% to 67.2%.
Perspective
Experiments cover mathematical reasoning, TACO coding and WebShop interactive decision-making, with Qwen2.5-1.5B, 3B, 7B and Llama-3.2-1B trained on MATH, trajectories of at most six turns, up to 2048 response tokens per turn and a three-second code execution timeout. The intended readers are reinforcement-learning researchers and engineers training tool-using language models.
The authors note that larger models, long-horizon software engineering, multi-agent coordination and variability across training seeds remain unevaluated; reward contrasts depend on sufficient sampled support, Supervised Replay depends on the quality of recorded calls, and convergence of the full regularized objective is left to theoretical work.