Self-distillation's teacher also needs teaching: TeachBack's bidirectional preferences lift Qwen3 math averages by up to 4.3 points

In on-policy self-distillation, a teacher that sees the reference solution can produce accurate trajectories that are still poorly suited to a student who sees only the problem, because direct access to a full solution shortens reasoning and skips exploration and verification. Earlier remedies mainly regulate how the student uses an already generated teacher signal, for example anchoring updates at high-risk tokens to a frozen policy, contrasting correct and incorrect hints, or removing a reference-only component; how the teacher itself is constructed remained an upstream question. In an audit of 256 DeepMath problems with four rollouts each and a 16K generation cap, the problem-only student reached 79.88% accuracy, while an instruction-style teacher given the full solution reached 99.12% accuracy but averaged only 1314.3 tokens, with exploration density of 6.57 events per 1K tokens and breadth of 57.58%.

Reframing a trailing slice of the reference reasoning as incomplete working notes, and asking the teacher to re-solve the problem from the start, keeps accuracy relatively high while reducing copying of the reference text. When the same 50% reference suffix was framed as an authoritative reference solution, explicit mention of the reference was 73.05% and strong attribution 45.80%; as working notes, explicit mention fell to 11.91% and strong attribution to 6.54%. Working notes reached 87.30% and 88.28% accuracy at 25% and 50% injection, above the 79.88% problem-only student; at 75% and 100% injection accuracy was 86.72% and 87.70%, while exploration density fell from 11.64 to 7.86.

Training samples paired teacher and student trajectories from the shared policy and cross-scores both responses under both prompts, yielding two preference objectives that point in opposite directions. The student-side objective prefers the teacher-origin response under the problem-only prompt, while the teacher-side objective prefers the student-origin response under the privileged prompt; the reversed preference does not claim the student answer is more accurate, but feeds student exploration and backtracking back into the role that generates future teacher targets. Both directional margins stayed positive during training, with the teacher-side margin growing much more, mainly because the teacher-origin response's score falls under the privileged prompt; with teacher-side weights of 0.5 and 1.0 the final held-out math macro mean@12 was 53.99% and 54.02%, versus 45.44% and 47.45% at 0.1 and 0.25.

At all three Qwen3 scales, average math and logical-reasoning scores exceed base, GRPO, OPSD, SPIN and PBSD. Relative to base, math averages improve by 4.3, 2.2 and 2.5 points and Knight-and-Knaves averages by 6.1, 3.9 and 7.5 points for Qwen3-1.7B, 4B and 8B; Qwen3-1.7B scores 33.0 at 11 roles, 1.0 point below base's 34.0. Math uses mean@12 on AMC23, AIME24 and AIME25, logic uses pass@1 on Knight-and-Knaves, training uses a hard DeepMath subset and 4–8-role logic problems, and held-out tests include 9–11 roles.

The demonstrated setting is Qwen3-1.7B, 4B and 8B trained on a hard DeepMath subset and evaluated on AMC23, AIME24, AIME25 and Knight-and-Knaves with 4–11 roles; cross-architecture results cover only math benchmarks for DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B. The intended readers are researchers and engineers working on reasoning post-training and self-distillation.

Exploration and reference-dependence metrics are surface-text proxies: polysemous markers such as wait, check, consider and approach can overcount the intended behavior, while paraphrased use of privileged information can evade exact phrase and n-gram matching. The teacher-side preference can also reinforce student errors or unproductive verbosity, so correctness, exploration, reference dependence, response length and downstream performance need joint evaluation. The theoretical analysis covers ordinary gradient steps on a fixed replay batch, not the full AdamW update used in practice, and does not guarantee improvement across rollout refreshes.