When every reward is zero, failed rollouts become teaching moments: live hints lift Llama-3.2-3B math by 78.3% on average
Synopsis
Reinforcement learning from verifiable rewards (RLVR) can stall when a jump in task difficulty makes every sampled trajectory fail, leaving zero reward and no learning signal. MENTOR keeps those failed trajectories, has a set of agents diagnose the error in real time and write a targeted instructional hint, then prepends the hint to the original problem for a new rollout. On math and code generation it beats GRPO, DAPO and static hint libraries, with average relative math gains up to 78.3% on Llama-3.2-3B-Instruct.
Interpretation
When a whole batch of rollouts fails and reward is zero, MENTOR keeps the samples and turns them into a teaching moment, so a stalled problem can produce non-zero reward again. Common practice filters out all-zero-reward batches or relies on pre-written hint libraries and curricula; MENTOR retains the failed trajectories and generates hints from them. On math and code benchmarks MENTOR exceeds GRPO, DAPO, Critique-GRPO, SRPO, LUFFY, QUESTA, HINT and Scaf-GRPO across five models, with a 78.3% average relative math gain on Llama-3.2-3B-Instruct.
Hints are not pre-written; they are generated for the specific cause of the current policy's failure and prepended to the original problem for resampling. Static hint libraries drift away from the policy as it updates, while MENTOR's diagnosis and hints follow the policy's evolving failure modes. An attribution agent outputs error keywords, a generation agent drafts a hint, a judgment agent scores it by rules plus LLM-as-a-Judge, and a repair agent rewrites it; in a case study a first-round hint judged too vague passed after rewriting.
The mechanism changes only the input of failed samples, leaving the GRPO loss, optimizer and reward mechanism untouched. It can therefore plug into existing GRPO-style training pipelines without rebuilding the training infrastructure. Training ran one epoch on eight H20-3E GPUs with four trajectories per problem and Qwen3-32B teaching agents; wall-clock time was about 24% faster than Critique-GRPO (17.9h versus 23.5h).
Perspective
Results cover mathematical reasoning (DeepMath-103K, 10K train/2.5K validation) and code generation (LiveCodeBench, 1,055 instances, 5:1 split), evaluated on Qwen2.5-Math-7B, DeepSeek-Coder-1.3B, DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-4B and Llama-3.2-3B-Instruct, with MMLU probing out-of-distribution behavior. It suits researchers and engineers working on RLVR training stability and hint-based intervention.
Prompt generation and reasoning guidance are validated only in single-modal settings; applicability to complex dynamic environments and multimodal fusion tasks has not been fully explored. Max Retry peaks at 2 for both math and code, while the best refinement count varies by model and task, 2 for Qwen3-4B on math and 3 for Llama-3.2-3B-Instruct.