Self-distillation's reasoning gains don't survive new problems: coding help stays inside the training set, and gradients often point away from policy gradients

Self-distillation gains on reasoning tasks do not reliably transfer to new problems: on coding, scores rise on problems seen during training but the advantage disappears on a fresh set of problems. The earlier report measured an overall gain on LiveCodeBench v6, where training and evaluation use the same set of coding problems, so generalization was not cleanly measured. After splitting problems into disjoint halves and training Qwen3-8B, Qwen3-14B and Olmo-3-7B-Instruct for 200 steps with four seeds each, policy-gradient training improves unseen problems by about 3.7, 5.2 and 2.6 points, while self-distillation changes them by about -2.5, -0.5 and -9.9 points.

In-domain gains on science reasoning and tool use turn into losses out of domain, while policy-gradient training largely preserves accuracy. The earlier report found self-distillation ahead of policy-gradient training in these domains but did not test cross-domain behavior. Training Qwen3-8B separately on five science-reasoning and tool-use domains gives similar in-domain averages (self-distillation versus policy gradient), yet self-distillation lowers average out-of-domain accuracy by roughly 1 to 5 points in four of five domains, while policy gradient raises it by roughly 0.5 to 2 points.

The math improvement comes from moving the teacher's thinking mode into the student, not from passing down privileged information such as a reference solution. The earlier report attributed the gain to reasoning transfer from a teacher conditioned on a reference solution, but teacher and student differ in two ways: whether the reference is visible and whether thinking is enabled. With the teacher's thinking mode fixed, swapping the reference solution for another problem's or removing it barely changes the teacher's token probabilities; enabling teacher thinking raises average thinking-off accuracy by about 3.6, 4.4, 5.6 and 6.4 points from Qwen3-1.7B to 14B, whereas changing the reference moves it by about 0.1 to 1.4 points.

Self-distillation updates do not align with the policy-gradient direction that directly optimizes expected reward on held-out problems, and often point the other way. Earlier work motivated self-distillation as approximate policy improvement on the grounds that its gradient aligns with the policy gradient. Across Qwen3 models, policy-gradient training has positive cosine similarity with the held-out policy gradient (about 0.02 to 0.10), while OPSD variants are negative (about -0.01 to -0.10) and SDPO also points away on Qwen3-8B.

The work examines two self-distillation recipes, SDPO and OPSD, across four verifiable task families: coding, science reasoning, tool use and math, with models limited to the Qwen3 family and Olmo-3-7B. It is aimed at researchers and engineers choosing a post-training recipe for reasoning or planning to feed privileged information into a training signal. The authors note that updates using harnesses, contrastive objectives, or more complex aggregation of feedback from neighboring rollouts are outside the scope and may address the observed limitations.

The authors state that conclusions are limited to the Qwen3 and Olmo model families and the SDPO and OPSD recipes, and do not cover newer self-distillation techniques; experiments also focus on verifiable domains, and the authors acknowledge self-distillation has uses in non-verifiable settings. On Olmo-3-7B-Instruct, policy-gradient and OPSD cosine similarities are positive but their jackknife errors are comparable to the values, so none are clearly resolved from zero.