# KL gradient–recovery trick

Multi-token prediction (MTP), which trains a language model to predict several future tokens, is now a common pre-training ingredient, but its standard cross-entropy gradient contains a low signal-to-noise residual that grows noisier with scale. Soft-MTP removes that residual and backpropagates only the KL self-distillation signal hidden inside it, reusing cross-entropy gradients already computed and materializing no vocabulary-sized logit matrices, as a one-line code change. On LLaMA-style models from 167M to 1.2B parameters, Soft-MTP beats standard MTP at every scale with lower next-token perplexity, faster self-speculative decoding and higher zero-shot accuracy; from 689M it also lowers MTP perplexity, and at 1.2B it gives stronger math and code reasoning after fine-tuning.

Multi-token prediction (MTP), which trains a language model to predict several future tokens, is now a common pre-training ingredient, but its standard cross-entropy gradient contains a low signal-to-noise residual that grows noisier with scale. Soft-MTP removes that residual and backpropagates only the KL self-distillation signal hidden inside it, reusing cross-entropy gradients already computed and materializing no vocabulary-sized logit matrices, as a one-line code change. On LLaMA-style models from 167M to 1.2B parameters, Soft-MTP beats standard MTP at every scale with lower next-token perplexity, faster self-speculative decoding and higher zero-shot accuracy; from 689M it also lowers MTP perplexity, and at 1.2B it gives stronger math and code reasoning after fine-tuning.