Raising Adam's second moment only for the output projection removes 39.4%–67.9% of catastrophic forgetting across eight settings from 160M to 12B
In data-free continual pre-training and fine-tuning, the authors use training-time parameter freezing to localize catastrophic forgetting to the output embedding rows of tokens rare in the new corpus, show that these rows receive persistent one-sided softmax gradients that Adam's second-moment normalization amplifies into full-sized updates, and raise Adam's second moment only for the output projection, removing 39.4%–67.9% of forgetting across seven stable settings from 160M to 12B and four model families without degrading target learning.