# Multi-turn agents

5 related articles

Subscribe to updates

arXiv Nov 4, 2026
Multi-turn RL pushes LLM agents past the base model only with roomy budgets, and the edge fades or reverses when turns tighten
Multi-turn agentic reinforcement learning is widely used to strengthen the long-horizon behavior of large language model agents, yet trajectory-level success rates hide what it does to individual decisions. Across four interactive tasks, DeepCoder-MT, BFCL, InterCode-SQL and FrozenLake, models trained with GRPO are evaluated along two axes, sampling breadth and interaction depth: with more sampled trajectories the RL model consistently beats its base model, but when the turns allowed per trajectory are tightened that edge shrinks, disappears, and on some tasks reverses. Matched-state comparisons show better next-step actions on only 46.8% of DeepCoder-MT states and 42.8% of FrozenLake states, with the gain concentrated in error recovery and later trajectory stages.
Multi-turn agentic reinforcement learning is widely used to strengthen the long-horizon behavior of large language model agents, yet trajectory-level success rates hide what it does to individual decisions. Across four interactive tasks, DeepCoder-MT, BFCL, InterCode-SQL and FrozenLake, models trained with GRPO are evaluated along two axes, sampling breadth and interaction depth: with more sampled trajectories the RL model consistently beats its base model, but when the turns allowed per trajectory are tightened that edge shrinks, disappears, and on some tasks reverses. Matched-state comparisons show better next-step actions on only 46.8% of DeepCoder-MT states and 42.8% of FrozenLake states, with the gain concentrated in error recovery and later trajectory stages.
Chinese Academy of Sciences Tsinghua University
Kejian Zhu（First author）
BudgetMix RL Agentic Reinforcement Learning LLM Agents Error recovery Multi-turn agents
arXiv Nov 10, 2026
Reward design for clinical multi-turn agents: outcome-only training shortens the work-up, dense question credit keeps high-risk cases
Clinical multi-turn agents must decide what to ask, when to stop, and what to do from incomplete information. Across three controlled simulated settings, medication refill triage, acute diagnostic work-up and dosing titration, the study compares reward designs: outcome-only GRPO often raises average value by asking fewer questions while producing premature commitment and weaker high-risk performance, whereas extra credit for questions that reduce uncertainty about the hidden state raises held-out value and moderate-case support across three training seeds in diagnostics without the high-risk seed collapse seen under outcome-only training.
Clinical multi-turn agents must decide what to ask, when to stop, and what to do from incomplete information. Across three controlled simulated settings, medication refill triage, acute diagnostic work-up and dosing titration, the study compares reward designs: outcome-only GRPO often raises average value by asking fewer questions while producing premature commitment and weaker high-risk performance, whereas extra credit for questions that reduce uncertainty about the hidden state raises held-out value and moderate-case support across three training seeds in diagnostics without the high-risk seed collapse seen under outcome-only training.
Brendan P. Murphy（First author）
Safe Reinforcement Learning GRPO Credit Assignment Multi-turn agents Reward Design
arXiv Oct 12, 2026
Multi-turn agent distillation stops feeding student reasoning to the teacher: a 4B student beats its 32B teacher on ALFWorld Seen at 43.39%
Multi-turn agent distillation now grounds teacher supervision in the interaction state, keeping only executed actions and observations, and the Qwen3-4B student surpasses its 32B teacher on ALFWorld Seen at 43.39%.
Alibaba Group (China)Shanghai Jiao Tong University Tsinghua University
Zhentao Zhang（First author）
State-OPD On-Policy Distillation Multi-turn agents ALFWorld Qwen3
arXiv Oct 1, 2026
PivotOPD concentrates distillation on pivotal mistakes and the turns after them, lifting Qwen3-1.7B by 5.5 points over the strongest baseline on ALFWorld
Analyzing failed ALFWorld rollouts of three Qwen3 models (8B–235B), the work finds that more than half of failures contain an early and usually recoverable pivotal mistake, and proposes PivotOPD: a teacher model locates pivotal turns and names a gold action plus recovery actions at the next few turns, preventive distillation uses reverse KL on the committed mistake while recovery distillation uses forward KL on self-teacher responses, both folded into a single PPO update with group-based RL, achieving the best average performance on ALFWorld, WebShop, and Search-based QA for Qwen3-1.7B and Qwen3-8B students and transferring to a Nemotron-3.5 student on SWE-Bench Verified (+3.2% resolve rate).
Analyzing failed ALFWorld rollouts of three Qwen3 models (8B–235B), the work finds that more than half of failures contain an early and usually recoverable pivotal mistake, and proposes PivotOPD: a teacher model locates pivotal turns and names a gold action plus recovery actions at the next few turns, preventive distillation uses reverse KL on the committed mistake while recovery distillation uses forward KL on self-teacher responses, both folded into a single PPO update with group-based RL, achieving the best average performance on ALFWorld, WebShop, and Search-based QA for Qwen3-1.7B and Qwen3-8B students and transferring to a Nemotron-3.5 student on SWE-Bench Verified (+3.2% resolve rate).
Princeton University Nvidia (United States)University of Maryland, College Park
Yinghui He（First author）
PivotOPD On-Policy Distillation Multi-turn agents ALFWorld
arXiv Sep 30, 2026
ActFirst-OPD lets multi-turn agents act before reasoning, speeding on-policy distillation training 2.3x, 1.8x and 4.9x on ALFWorld, WebShop and ScienceWorld
The work proposes ActFirst-OPD, which decouples environment interaction from full-response generation: the student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, switches to autonomous next-action prediction once the transition deviates from the reference trajectory, and asynchronously generates full think-then-act responses from the collected interaction contexts for token-level teacher supervision; across 0.6B, 1.7B and 4B Qwen3 students it achieves average wall-clock training speedups of 2.3x on ALFWorld, 1.8x on WebShop and 4.9x on ScienceWorld over Vanilla OPD, while matching or exceeding mean task success rate in eight of nine benchmark-model settings.
The work proposes ActFirst-OPD, which decouples environment interaction from full-response generation: the student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, switches to autonomous next-action prediction once the transition deviates from the reference trajectory, and asynchronously generates full think-then-act responses from the collected interaction contexts for token-level teacher supervision; across 0.6B, 1.7B and 4B Qwen3 students it achieves average wall-clock training speedups of 2.3x on ALFWorld, 1.8x on WebShop and 4.9x on ScienceWorld over Vanilla OPD, while matching or exceeding mean task success rate in eight of nine benchmark-model settings.
Hong Kong Polytechnic University Nanyang Technological University Southern University of Science and Technology
Zubin Zheng（First author）
ActFirst-OPD Reference-Conditioned Inverse Dynamics On-Policy Distillation Multi-turn agents Asynchronous Full-Response Generation

You’re all caught up

Multi-turn RL pushes LLM agents past the base model only with roomy budgets, and the edge fades or reverses when turns tighten

Reward design for clinical multi-turn agents: outcome-only training shortens the work-up, dense question credit keeps high-risk cases

Multi-turn agent distillation stops feeding student reasoning to the teacher: a 4B student beats its 32B teacher on ALFWorld Seen at 43.39%

PivotOPD concentrates distillation on pivotal mistakes and the turns after them, lifting Qwen3-1.7B by 5.5 points over the strongest baseline on ALFWorld

ActFirst-OPD lets multi-turn agents act before reasoning, speeding on-policy distillation training 2.3x, 1.8x and 4.9x on ALFWorld, WebShop and ScienceWorld