# Reinforcement Learning from Verifiable Rewards

Offline demonstrations, current-policy rollouts, and historical rollouts can now be trained under one regression target, and on four math-reasoning benchmarks the unified configuration improves Qwen3-1.7B benchmark-averaged avg@32 and pass@32 over two-stage SFT plus DAPO by 2.33 and 4.44 percentage points.

Offline demonstrations, current-policy rollouts, and historical rollouts can now be trained under one regression target, and on four math-reasoning benchmarks the unified configuration improves Qwen3-1.7B benchmark-averaged avg@32 and pass@32 over two-stage SFT plus DAPO by 2.33 and 4.44 percentage points.

Reinforcement learning from verifiable rewards (RLVR) can stall when a jump in task difficulty makes every sampled trajectory fail, leaving zero reward and no learning signal. MENTOR keeps those failed trajectories, has a set of agents diagnose the error in real time and write a targeted instructional hint, then prepends the hint to the original problem for a new rollout. On math and code generation it beats GRPO, DAPO and static hint libraries, with average relative math gains up to 78.3% on Llama-3.2-3B-Instruct.

Reinforcement learning from verifiable rewards (RLVR) can stall when a jump in task difficulty makes every sampled trajectory fail, leaving zero reward and no learning signal. MENTOR keeps those failed trajectories, has a set of agents diagnose the error in real time and write a targeted instructional hint, then prepends the hint to the original problem for a new rollout. On math and code generation it beats GRPO, DAPO and static hint libraries, with average relative math gains up to 78.3% on Llama-3.2-3B-Instruct.

The work introduces Lexicographic Multi-Objective On-Policy Distillation (LMOPD), which for each student rollout selects the specialist for the first objective whose gate detects a deficiency and locally projects its centered log-policy correction to remove components opposing higher-priority specialists; on three math benchmarks with 30B-A3B mixture-of-experts models, the two-expert setting fully retains accuracy and reasoning-quality gains in point estimates while acquiring 46.9% of the conciseness gain, and the four-expert setting retains about 90% of both the accuracy gain and reasoning-correctness gain versus only about 57% for the next best evaluated baseline.

The work introduces Lexicographic Multi-Objective On-Policy Distillation (LMOPD), which for each student rollout selects the specialist for the first objective whose gate detects a deficiency and locally projects its centered log-policy correction to remove components opposing higher-priority specialists; on three math benchmarks with 30B-A3B mixture-of-experts models, the two-expert setting fully retains accuracy and reasoning-quality gains in point estimates while acquiring 46.9% of the conciseness gain, and the four-expert setting retains about 90% of both the accuracy gain and reasoning-correctness gain versus only about 57% for the next best evaluated baseline.