Skip to main content

# Active Taskless Distillation

2 related articles

Subscribe to updates

arXiv Sep 29, 2026
Peking University-led team distills coding ability from one-word answers: a Qwen2.5-1.5B student beats an exact nuisance-matched control by 5.34 points on HumanEval+
The work introduces Active Taskless Distillation (ATD), which uses a public ancestor model to select unrelated prompts on which it is nearly indifferent between two ordinary words, has a privately post-trained teacher return a single word per prompt, and trains a same-ancestor student only on those prompt-word pairs; in the primary Qwen2.5-1.5B coding experiment, 5,664 single-word responses yield a 5.34 percentage-point gain on HumanEval+ over an exact nuisance-matched control, with positive mean gains also observed in scientific knowledge, commonsense reasoning, and reading comprehension across additional model generations, sizes, and families.
The work introduces Active Taskless Distillation (ATD), which uses a public ancestor model to select unrelated prompts on which it is nearly indifferent between two ordinary words, has a privately post-trained teacher return a single word per prompt, and trains a same-ancestor student only on those prompt-word pairs; in the primary Qwen2.5-1.5B coding experiment, 5,664 single-word responses yield a 5.34 percentage-point gain on HumanEval+ over an exact nuisance-matched control, with positive mean gains also observed in scientific knowledge, commonsense reasoning, and reading comprehension across additional model generations, sizes, and families.
Georgia Institute of Technology Peking University ShanghaiTech University Tsinghua University
Ziyang Zhang（First author）
Active Taskless Distillation Behavioral Shadow of Post-Training Capability Transfer
arXiv Sep 26, 2026
Post-training capabilities leak through unrelated text: one teacher word per prompt yields a 5.34 pp gain on HumanEval+
The authors introduce Active Taskless Distillation (ATD), which selects prompts where the teacher and student's shared public ancestor is nearly indifferent between two ordinary words, and lets a student initialized from that ancestor learn solely from the resulting prompt-word pairs, without target-task examples, teacher logits, or teacher parameters; in the primary coding experiment with Qwen2.5-1.5B, 5,664 samples yield a 5.34 pp gain on HumanEval+ over an exact nuisance-matched control, with transfer also shown in scientific knowledge, commonsense reasoning, and reading comprehension across additional model generations, sizes, and families, and functional analyses indicating the learned signal is composable and tracks the teacher's update strength.
The authors introduce Active Taskless Distillation (ATD), which selects prompts where the teacher and student's shared public ancestor is nearly indifferent between two ordinary words, and lets a student initialized from that ancestor learn solely from the resulting prompt-word pairs, without target-task examples, teacher logits, or teacher parameters; in the primary coding experiment with Qwen2.5-1.5B, 5,664 samples yield a 5.34 pp gain on HumanEval+ over an exact nuisance-matched control, with transfer also shown in scientific knowledge, commonsense reasoning, and reading comprehension across additional model generations, sizes, and families, and functional analyses indicating the learned signal is composable and tracks the teacher's update strength.
Georgia Institute of Technology Peking University ShanghaiTech University Tsinghua University
Ziyang Zhang（First author）
Code Generation Knowledge Distillation Active Taskless Distillation Subliminal Learning

You’re all caught up