Distillation Rollouts Stop Copying the Student: Qwen3-4B Math Average Gains 6.91 Points and Generation Length Drops 41%
Synopsis
On-policy distillation no longer treats the student's own samples as the only rollout distribution: previously each round sampled from the current student, forcing the teacher to supervise token by token on trajectories the student reaches but the teacher rarely does, so that supervision becomes noisy and degenerate patterns such as repetition are sampled again in later rounds. The new method reweights by relative teacher support, derives a geometric mixture of student and teacher inside a KL trust region, and trains a proposal policy to approximate that mixture in one forward pass. Across 16 math, knowledge QA and medical reasoning benchmarks, Qwen3-4B-Base gains 6.91 average accuracy over OPD on math while generating 41% shorter responses.
Interpretation
Distillation training, where the student's own samples receive token-level teacher supervision, now shifts those sampled trajectories toward regions the teacher can reach more easily. The standard recipe fixes each round's rollout policy to the current student, so the teacher must supervise trajectories that are low-probability for the student and rarely generated by the teacher, and noise and repetition patterns accumulate across rounds. When the teacher completed both a student trajectory and its own trajectory from the same prompt during training, semantic similarity between the two completions drifted downward and the fraction of prompts where the teacher's student-trajectory completion received a lower score from a process reward model that grades reasoning steps rose; with the new rollout, 10th-percentile similarity is 0.30 higher and the lower-score fraction is 18 points lower than OPD.
The target rollout distribution is a geometric mixture of student and teacher, with the mixing strength set by relative teacher support. Relative support is the log ratio of teacher to student probability for the same token at the same prefix; maximizing its expectation inside a KL-defined local trust region yields a geometric mixture that multiplies student and teacher probabilities by powers and renormalizes, with one scalar interpolating between pure student sampling and pure teacher sampling, and a proposal policy initialized from the current student approximating the mixture so generation needs one forward pass. Evaluation covers Qwen3-1.7B-Base and Qwen3-4B-Base on math, with knowledge QA and medical reasoning on Qwen3-4B-Base, for 16 benchmarks in total.
Perspective
The demonstrated scope is on-policy distillation within the Qwen3 family: the main experiments use Qwen3-4B-Instruct-2507 as teacher and Qwen3-1.7B-Base and Qwen3-4B-Base as students, covering math, knowledge QA and medical reasoning, with an additional math extension to an 8B student.
The proposal policy only approximates the tilted target, and the geometric-mixture and trust-region interpretation applies to the ideal target; the tilt strength is treated as a fixed scalar rather than adapted during training, and evaluation is concentrated in the Qwen3 family.