# Weak-to-Strong Transfer

The work shows that Direct-OPD's token-level log-ratio reward measures only relative change and can stay fixed while the probability mass that both checkpoints assign to the student's candidate tokens vanishes, whereas the teacher–reference JSD and both KL directions vanish with that mass; it therefore proposes S²D-OPD, which ranks student-sampled states by teacher–reference JSD and masks Direct-OPD supervision at low-divergence states, retaining only the top 10% per response, improving held-out accuracy over dense Direct-OPD on AIME and HMMT benchmarks in seven of eight settings across two teacher pairs and four student models from 1.7B to 8B parameters and matching it in the eighth, without extra forward passes.

The work shows that Direct-OPD's token-level log-ratio reward measures only relative change and can stay fixed while the probability mass that both checkpoints assign to the student's candidate tokens vanishes, whereas the teacher–reference JSD and both KL directions vanish with that mass; it therefore proposes S²D-OPD, which ranks student-sampled states by teacher–reference JSD and masks Direct-OPD supervision at low-divergence states, retaining only the top 10% per response, improving held-out accuracy over dense Direct-OPD on AIME and HMMT benchmarks in seven of eight settings across two teacher pairs and four student models from 1.7B to 8B parameters and matching it in the eighth, without extra forward passes.

The study systematically characterizes the scaling properties of same-family on-policy distillation (OPD) on Qwen2.5 (0.5B–14B) math reasoning, finding a regular early useful-transfer regime in which held-out accuracy rises approximately linearly with the square root of token-level reverse KL from the student initialization, and fitting power laws in student scale, teacher scale, and teacher gold score that predict peak accuracy and transfer rate, with every weak-to-strong student peaking above its own teacher.

The study systematically characterizes the scaling properties of same-family on-policy distillation (OPD) on Qwen2.5 (0.5B–14B) math reasoning, finding a regular early useful-transfer regime in which held-out accuracy rises approximately linearly with the square root of token-level reverse KL from the student initialization, and fitting power laws in student scale, teacher scale, and teacher gold score that predict peak accuracy and transfer rate, with every weak-to-strong student peaking above its own teacher.

The work introduces Allspark, a training and inference framework in which a weak teacher and a frozen copy of the same model alternate reasoning segments during training while only the teacher is updated from weak-model rollouts, and at inference a stronger frozen student replaces the frozen partner; across Qwen3-1.7B/4B math and reasoning tasks and Inkling-Small teacher evaluations on ARC-AGI-2 with Inkling, Kimi-K2.6, and Nemotron-3-Ultra students, accuracy gains are observed along with an explicit accuracy-token tradeoff.

The work introduces Allspark, a training and inference framework in which a weak teacher and a frozen copy of the same model alternate reasoning segments during training while only the teacher is updated from weak-model rollouts, and at inference a stronger frozen student replaces the frozen partner; across Qwen3-1.7B/4B math and reasoning tasks and Inkling-Small teacher evaluations on ARC-AGI-2 with Inkling, Kimi-K2.6, and Nemotron-3-Ultra students, accuracy gains are observed along with an explicit accuracy-token tradeoff.