# Mixture Proposal On-Policy Distillation

On-policy distillation no longer treats the student's own samples as the only rollout distribution: previously each round sampled from the current student, forcing the teacher to supervise token by token on trajectories the student reaches but the teacher rarely does, so that supervision becomes noisy and degenerate patterns such as repetition are sampled again in later rounds. The new method reweights by relative teacher support, derives a geometric mixture of student and teacher inside a KL trust region, and trains a proposal policy to approximate that mixture in one forward pass. Across 16 math, knowledge QA and medical reasoning benchmarks, Qwen3-4B-Base gains 6.91 average accuracy over OPD on math while generating 41% shorter responses.

On-policy distillation no longer treats the student's own samples as the only rollout distribution: previously each round sampled from the current student, forcing the teacher to supervise token by token on trajectories the student reaches but the teacher rarely does, so that supervision becomes noisy and degenerate patterns such as repetition are sampled again in later rounds. The new method reweights by relative teacher support, derives a geometric mixture of student and teacher inside a KL trust region, and trains a proposal policy to approximate that mixture in one forward pass. Across 16 math, knowledge QA and medical reasoning benchmarks, Qwen3-4B-Base gains 6.91 average accuracy over OPD on math while generating 41% shorter responses.