Agent training stops grading only the ending: STEP-VERIFIER scores each step from prefix splits in logged runs
Synopsis
Tool-using language-model agents often need dozens of interactions before a single success-or-failure label arrives, leaving no way to tell which step went wrong. STEP-VERIFIER turns a success-labeled corpus of logged interactions into action-level comparisons: the difference in average final return between two continuations that share a common prefix labels which of two actions was better, and a Bradley-Terry ranking head converts those pairs into calibrated per-step scores. On Llama-3.1-8B-Instruct and Mistral-7B-Instruct backbones, it beats outcome-only GRPO on the average metric in all six benchmark-backbone combinations across GAIA, ToolBench and AgentBoard, and its per-action scores agree with human judgments on 81.5% of pairs.
Interpretation
An agent's intermediate actions can now be scored individually before an episode ends, instead of waiting for a whole-trajectory success label. Training signals previously indicated only whether an entire attempt succeeded, so an episode that failed because of one wrong call at step two was indistinguishable to the learner from one that failed everywhere. With Llama-3.1-8B-Instruct, GAIA average accuracy rises from 21.5% under GRPO to 30.1%, ToolBench average win rate from 30.4% to 38.9%, and AgentBoard success rate from 24.8% to 35.4%.
The scorer reads the current environment observation, a candidate action and a truncated window of recent interaction, and outputs a scalar score. Supervision comes from two branches in the logs that share a common prefix, using the difference in their continuations' average final return as the preference label for that action pair, with no human step annotations and no extra environment interaction. A Bradley-Terry ranking head turns pairwise comparisons into score gaps; on 2,000 sampled observation-action pairs its pairwise judgments agree with three annotators on 81.5% of comparisons, versus 62.4% for the trajectory ORM and 73.2% for the binary step PRM.
During training the scorer first filters low-quality logs, then injects per-step feedback into the advantage with linear annealing, and once annealed to zero the deployed policy never queries it. Holding the feedback weight constant yields only 26.5% GAIA average, below the 28.3% obtained without dense feedback at all, showing that a signal which never exits hurts late training. Defaults give 30.1% GAIA average, 38.9% ToolBench average and 35.4% AgentBoard success rate; the final policy evaluated alone matches its training-time numbers, and adding test-time filtering changes results by at most 0.1 points.
Perspective
The demonstrated scope is training and evaluation with Llama-3.1-8B-Instruct and Mistral-7B-Instruct policy backbones on three public agent benchmarks, GAIA, ToolBench and AgentBoard, with a default Llama-3.2-3B-Instruct verifier reading a 1,024-token interaction window.
The authors note that the value gaps are those of the logging policy's continuations, not counterfactual gaps under an improved policy, and that finite-support pairs produce noisy labels whose deviation from the population optimum decays as the minimum support grows. They also note that hard, rarely visited states recur less in the logs, so the verifier is least informed exactly there, and they list targeted exploration of uncertain prefixes or cross-task transfer of verifier scores as the natural continuation.