When students write wrong, teachers drift too: SPTA first teaches the teacher to continue from a student's half-finished reasoning
Synopsis
On-policy distillation trains a student on trajectories the student itself generates, but student prefixes can carry reasoning errors or patterns unfamiliar to the teacher, and the teacher's token-level supervision becomes less reliable as prefixes grow longer. Before distillation begins, SPTA trains the teacher on student prefixes of random lengths, rewarding correct final answers, so the teacher learns to recover from student-induced states rather than imitate the student's reasoning; standard on-policy distillation then proceeds with the distillation objective unchanged. On mathematical reasoning benchmarks, students trained with SPTA average better than under standard on-policy distillation and better than under on-policy distillation with task-level RLVR-adapted teachers.
Interpretation
On-policy distillation shifts the training object from trajectories the teacher generates to trajectories the student actually walks, so the teacher must supervise continuations of the student's half-finished reasoning. Student-generated prefixes may contain reasoning errors or follow patterns unfamiliar to the teacher, which undermines the reliability of token-level supervision, and this degradation becomes more pronounced as prefixes grow longer. On mathematical reasoning benchmarks, average student performance with SPTA exceeds standard on-policy distillation and on-policy distillation with task-level RLVR-adapted teachers.
The approach reshapes the teacher before distillation starts, so it is used to the states on which it will be queried. Student rollout prefixes of random lengths are collected, and the teacher is continually trained to continue from them using reinforcement learning with verifiable rewards (RLVR, which rewards correct final answers). Treating each prefix as context and rewarding only correct final answers teaches the teacher to recover from student-induced states rather than imitate the student's reasoning; the adapted teacher then runs standard on-policy distillation with no change to the distillation objective.
Perspective
The demonstrated scope is an average student-performance comparison on mathematical reasoning benchmarks against standard on-policy distillation and on-policy distillation with task-level RLVR-adapted teachers; the adapted teacher is then used for standard on-policy distillation with the distillation objective unchanged.
The material is a summary and gives no specific benchmark names, model sizes or improvement magnitudes; the claim that degradation worsens with longer prefixes is not quantified.