Reward design for clinical multi-turn agents: outcome-only training shortens the work-up, dense question credit keeps high-risk cases
Synopsis
Clinical multi-turn agents must decide what to ask, when to stop, and what to do from incomplete information. Across three controlled simulated settings, medication refill triage, acute diagnostic work-up and dosing titration, the study compares reward designs: outcome-only GRPO often raises average value by asking fewer questions while producing premature commitment and weaker high-risk performance, whereas extra credit for questions that reduce uncertainty about the hidden state raises held-out value and moderate-case support across three training seeds in diagnostics without the high-risk seed collapse seen under outcome-only training.
Interpretation
The training target for clinical multi-turn agents moves from the final action alone to how the agent gathers evidence. Earlier medical reinforcement learning evaluations mostly stressed whether the final answer was correct, while clinical decisions often unfold as a sequence of questions, interpretations and actions. In the acute diagnostics task, outcome-only GRPO raised overall value above the baseline level but lowered both high-risk value and high-risk support, and mean questions fell well below the exact adaptive oracle level.
Extra credit for questions that reduce uncertainty about the hidden clinical state changes the learned work-up. Under outcome-only scoring, the model is not told which question in a trajectory mattered and which was wasted. Across three training seeds in diagnostics, this dense credit raised overall value above the outcome-only level, increased moderate-case support, and kept high-risk support in all three seeds, whereas one outcome-only seed lost that case.
Splitting exploration from exploitation pushes high-risk coverage higher, at the cost of over-questioning low-acuity cases. Dense credit still requires one trajectory to balance asking more against committing early. In diagnostics, the decoupled method at its best checkpoint raised high-risk value and high-risk support above the dense-credit level, but mean questions rose above the baseline, the dense-credit policy and the oracle, and benign-case support fell.
Perspective
The conclusions rest on three synthetic, stylized simulated environments labelled by deterministic rules, with deterministic patient answers and binary findings; the authors state these are not clinical validation benchmarks and that trained policies should not be read as suitable for patient care.
The authors note that high-risk cases are few in the main held-out sets, only two for diagnostics, and that the decoupled result relies on a checkpoint selected on the held-out set; apart from the main diagnostics comparison, the decoupled method, the titration credit-coefficient sweep, the refill experiments and the biased-supervision repair experiments each use one training seed and are treated as exploratory. The authors also state that the information credit is used as an auxiliary process reward and do not claim it is policy invariant under every task and training configuration.