The distillation temperature everyone left at 1: raise it and forward KL overtakes reverse KL, adding 4.29 average ROUGE-L on GPT-2 Base
Synopsis
LLM distillation has largely kept the temperature at τ=1 and concluded that reverse KL beats forward KL. A gradient analysis shows temperature changes what forward KL learns but mainly rescales reverse KL gradients; across GPT-2 and OPT instruction-following benchmarks reverse KL leads at τ=1, forward KL overtakes at higher temperatures, and in on-policy GPT-2 Base distillation forward KL rises from 16.95 to 21.24 average ROUGE-L.
Interpretation
Which KL direction wins in LLM distillation depends on the distillation temperature rather than being a fixed verdict. Earlier comparisons were mostly run at the default τ=1 and concluded reverse KL was superior. In on-policy GPT-2 XL to GPT-2 Base distillation, reverse KL reaches 19.97 average ROUGE-L and forward KL 16.95 at τ=1; forward KL rises to 21.24 at τ=3.5, while reverse KL gains only 0.44 before degrading at higher temperatures.
Temperature acts differently on the two objectives: it changes the teacher distribution forward KL matches, while for reverse KL it mainly rescales gradients. Forward KL targets the softened teacher probabilities, and temperature moves probability mass from high-logit tokens to lower-probability ones; reverse KL gradients are driven by student-weighted teacher-student logit gaps. As temperature grows both gradients reduce to teacher-student logit matching; on GPT-2 Base forward KL improves from 19.41 to 20.97 at τ=3.0, GPT-2 Medium from around 19 to above 22, and OPT-1.3B from below 21 to above 24, with best results around τ=2.5 to 3.5.
Temperature helps not only forward KL but a family of KL-based objectives including symmetric KL, JS, SFKL and AKL, and changes their ranking. The ordering of objectives measured at the default τ=1 no longer holds once temperature is applied. On GPT-2 Base forward KL rises from 19.70 to 25.53 and AKL gains the most at 6.62; on OPT-1.3B gains range from 3.46 for reverse KL to 4.88 for AKL, and AKL with temperature reaches the best score of 29.62.
Perspective
The results come from off-policy and on-policy distillation on English instruction-following tasks, with teachers up to 7B parameters and LoRA fine-tuning for the Llama3.2 and Qwen3 pairs; the authors list larger teachers, other languages and task types as future work.
The authors note that compute limits kept teachers at 7B, that the Llama3.2 and Qwen3 pairs use LoRA rather than full fine-tuning, and that evaluation centers on English instruction following; gains are weaker and need careful tuning for reverse-KL-based objectives.