Distillation stops needing a stronger teacher: task-trained policy differences transfer across models, scale gaps barely do
Synopsis
Whether one policy difference transfers depends on the target: Qwen3 targets gain positively while Qwen2.5 targets under a Qwen3 reference gain almost nothing, and swapping in a Qwen2.5-3B reference turns those gains positive.
Interpretation
Transferring a policy difference no longer requires the teacher to be stronger than the student, only that the teacher improves on its own reference model. Conventional on-policy distillation pushes the student toward the teacher's output distribution, so it needs a substantially stronger teacher; when the two are comparable or the teacher is weaker, gains are limited and the student's existing capabilities may even be impaired. Reading the log-probability ratio between a proxy expert and a proxy reference as an implicit reward for KL-regularized policy optimization recovers conventional on-policy distillation as a special case and shifts the transferred object from the teacher's absolute distribution to the policy change induced by its post-training.
What decides transfer is whether the policy difference encodes a task direction, not how large the difference is. Relative policy transfer had been validated mainly on a narrow class of source pairs where the expert comes from reinforcement learning applied to the reference model, leaving other sources of policy difference unclear. With Qwen3-8B as the target, reinforcement-learning and supervised-fine-tuning differences both improve the target consistently, while four scale-only pairs yield much smaller gains; a task-optimized pair and a scale-only pair induce updates of comparable magnitude yet produce substantially different target gains.
The expert supplies the capability to be transferred, while the reference filters out behavior the two share and leaves only the residual. Prior work largely demonstrated transfer gains, with limited analysis of how the expert, reference and target jointly determine the outcome. With the expert fixed to Qwen3-4B-RL, replacing the Qwen3-4B reference with an early reinforcement-learning checkpoint cuts the proxy gain from 0.0484 to 0.0000, while using Qwen2.5-3B as the reference gives the largest target gain of 1.2307 despite the largest proxy gain.
One useful policy difference does not work for every target; whether the target can follow and generalize is what matters. Earlier studies mostly evaluated specific pairs on students from the same model generation and at similar scales, leaving the boundaries of transfer across different priors or capability levels unclear. Four Qwen3 proxy pairs applied to six targets give positive gains on Qwen3 targets, near-zero gains on Qwen2.5 targets under a Qwen3 reference, and positive gains on Qwen2.5 targets once the reference is switched to Qwen2.5-3B while Qwen3 targets stay successful.
Perspective
The report covers mathematical reasoning and code generation, with models drawn from the Qwen3 and Qwen2.5 families and evaluation on eight math benchmarks and three code benchmarks. The source describes the formulation as any-to-any policy transfer and states that conventional on-policy distillation is recovered as a special case.
The source states that transfer failure is not an intrinsic limitation of the target or the useful difference; instead the reference determines the form in which the difference is presented to the target, and matching the model family is one effective choice in the experiments. The source also notes that gains on code are generally smaller than on math, while the signs and relative ordering of the tested configurations stay consistent across the two domains.