Teacher supervision stops being copied token by token: reward-weighted distillation lifts math benchmarks by up to 6.5 points
Synopsis
Distillation no longer treats every teacher token alike: each token's weight is set by its predicted effect on the student's post-update reward. Earlier token-selection methods relied on proxies such as uncertainty or teacher-student divergence, which need not identify reward-improving supervision. RAD scores each token by differentiating through a temporary optimizer step, then updates token weights by projected ascent, improving mean accuracy on six competition-math benchmarks by 6.5 points for Qwen3-1.7B and 2.5 points for Qwen3-4B-Instruct-2507 at a matched training-response budget, and reaching 54.69% and 58.67% pass@1 on LiveCodeBench-v6.
Interpretation
Each distillation token can now be strengthened, suppressed, or reversed according to its effect on reward, instead of all teacher supervision being accepted as given. Standard on-policy distillation applies teacher supervision to every token regardless of whether the teacher's preference improves task reward, and existing token-selection methods rely on proxies such as uncertainty or teacher-student divergence that need not identify reward-improving supervision. At a matched training-response budget, mean accuracy on six competition-math benchmarks improves over the strongest baselines by 6.5 percentage points for Qwen3-1.7B (non-thinking) and 2.5 points for Qwen3-4B-Instruct-2507.
The weights come from differentiating through a temporary optimizer step to score each token's supervision, then updating token weights by projected ascent from the initial weights. This bilevel framework ties distillation-signal weights to the student's post-update reward rather than to uncertainty or teacher-student divergence. On LiveCodeBench-v6, Qwen3-1.7B (non-thinking) and OLMo3-7B-Instruct reach 54.69% and 58.67% pass@1, exceeding the strongest baselines by 8.7 and 6.4 points.
The simplified variant RAD-Simple achieves comparable math accuracy at the same training-response budget. The full method distinguishes each token's supervision contribution, while the simplified variant offers another path at an equal budget. Math accuracy is comparable to full RAD, again compared at the same training-response budget.
Perspective
This report is limited to summary-level material covering six competition-math benchmarks and LiveCodeBench-v6, with Qwen3-1.7B (non-thinking), Qwen3-4B-Instruct-2507, and OLMo3-7B-Instruct, all compared at a matched training-response budget. It is intended for researchers and engineers working on LLM post-training and distillation.
The material is summary-level and does not give the specific hyperparameters of the weight update, the convergence behavior of projected ascent, or how RAD-Simple differs from the full method on tasks beyond math. The cross-family and cross-task conclusions come from the two reported model families and two task types, and do not yet cover more settings.