Reward the search, not just the answer: process supervision lifts retrieval recall and holds up out of domain
Synopsis
Search agents for multi-hop information seeking are trained on step-level judgments of each search action instead of only on final-answer correctness. RAG-Gym casts search as a Markov decision process and compares architecture, parameter optimization and action evaluation, combining a reasoning-reflection architecture, preference optimization and critic-guided inference into Re2Search++. Trained on HotpotQA, it beats outcome-supervised baselines on the unseen Bamboogle benchmark with higher retrieval recall at comparable or fewer search steps, and a critic trained on Llama-3.1-8B raises GPT-4o-mini MedQA accuracy.
Interpretation
The training signal for search agents moves from final-answer correctness to step-by-step evaluation of intermediate search actions, so the agent keeps searching well beyond its training domain. The prior mainstream approach was outcome supervision, rewarding only correct final answers, which lets agents guess from parametric memory and perform superficial searches that satisfy the training loop. Trained on HotpotQA and evaluated on the unseen Bamboogle benchmark, process-supervised Re2Search++ outperforms the outcome-supervised Search-R1 and R1-Searcher while matching or exceeding them on their own in-domain tasks.
At each step the agent drafts a reasoning chain, checks which claim is not supported by retrieved evidence, and turns the first gap into the next query. ReAct and Search-o1 issue queries from topical association without an explicit check tied to answer reasoning. Across HotpotQA, 2WikiMultihopQA, Bamboogle and MedQA, the Re2Search architecture with the reasoning-reflection module reaches the highest performance under every training regime from zero-shot learning to PPO.
The gain comes mainly from better search queries rather than from extra answer optimization. Outcome-supervised agents often reach correct answers from ungrounded predictions, with low retrieval recall. On both Llama-3.1 and Qwen-2.5 backbones, Re2Search++ recalls the highest fraction of ground-truth supporting documents while issuing comparable or fewer search queries than baselines.
A critic trained on trajectories from a small open-source model can guide a different model family without access to its weights. Critics were previously tied to a specific backbone. A critic trained solely on Llama-3.1-8B trajectories raises GPT-4o-mini accuracy on MedQA from 76.8% to 78.1%.
Perspective
The demonstrated scope is four knowledge-intensive question-answering benchmarks: three Wikipedia-based multi-hop datasets and one medical multiple-choice dataset, with retrieval corpora and retrieval hyperparameters fixed across agents. The authors list open-ended search tasks, other domains and retrieval environments, and broader agentic tasks such as tool use and planning as future directions.
Process supervision depends on high-quality reward signals, and rollout-based rewards produce false positives in retrieval when an answer is coincidentally correct, so stronger model-based annotators work better but may add bias and cost. The three design dimensions cover representative rather than exhaustive candidate sets, and critic-guided Best-of-N inference adds computational cost.