Reasoning training stalls when every rollout scores zero: MENTOR turns failed attempts into live hints, lifting Llama-3.2-3B math by 78.3% on average
Lead
When every sampled trajectory for a problem scores zero and the learning signal vanishes, MENTOR diagnoses the failure inside the training loop and generates a targeted hint so the model earns non-zero reward again; on math and code generation it raises Llama-3.2-3B-Instruct's average math score by 78.3% relative.
Story
Reinforcement learning training can stall in a specific way: for one problem, every sampled reasoning trajectory is judged wrong, the whole batch earns zero reward, the gradient signal disappears, and the policy stops improving. Earlier responses either discard these all-zero batches outright or rely on pre-written hint libraries and pre-defined curricula, which drift away from the model's current errors as the policy changes. On math and code generation benchmarks, MENTOR scores above eight comparison methods across four models, for example a 78.3% average relative math gain on Llama-3.2-3B-Instruct and 35.2% on math plus 43.6% on code for DeepSeek-R1-Distill-Qwen-1.5B.
It treats failed trajectories as diagnostic material: first it locates the conceptual error the model keeps repeating, then writes a hint that points at the direction without giving the answer, and prepends that hint to the original problem for a fresh rollout. Four roles split the work: an Attribution Agent emits diagnostic keywords, a Generation Agent writes a candidate hint from them, a Judgment Agent scores quality by rules and model judgment, and a Repair Agent rewrites the hint until it passes or falls back to a safe default. Training runs on eight H20-3E GPUs, with DeepMath-103K for math and LiveCodeBench for code, four sampled trajectories per problem, and Qwen3-32B powering hint generation and judgment; ablations show accuracy drops of 4.50% without the Attribution Agent, 4.67% without the Generation Agent, and 2.28% without the Judgment-Repair loop.
What to watch
The mechanism is validated on text tasks in mathematical reasoning and code generation, covering math-specialized, code-specialized, lightweight distilled, and general-purpose models, trained on DeepMath-103K and LiveCodeBench with Qwen3-32B powering hint generation and judgment. It plugs into GRPO-style pipelines and modifies only the input data of collapsed samples, leaving the loss function, optimizer, and reward mechanism unchanged. The authors state in the limitations that the current prompt generation and reasoning guidance mechanism is validated only in single-modal scenarios, while applicability in complex dynamic environments and multi-modal fusion tasks has not been fully explored.
The authors confine the current mechanism to single-modal scenarios, leaving applicability in complex dynamic environments and multi-modal fusion tasks not fully explored. Hyperparameter analysis shows Max Retry peaks at 2 for both math and code, while the optimal Max Refine varies by model and task, for example Qwen3-4B peaks at 2 for math while the smaller Llama-3.2-3B-Instruct benefits from 3, indicating subtler error patterns need deeper semantic analysis. Case studies also include a guided rollout that still answers incorrectly, such as one new rollout that concludes No.