Ran Tian

Training-loss spikes have been hard to reproduce under controlled conditions, and their cause is not fully understood. Researchers show that simple quadratic models with heavy-tailed, power-law inputs reproduce the spikes, then align per-layer training dynamics of deep networks with such models and give a lightweight simulation algorithm. In a 1.3B-parameter Transformer language model case study, where an AdamW hyperparameter was changed to induce spikes, the simulation from a step-21k checkpoint predicts bumps for the Spiky1 and Spiky2 configurations while Normal stays comparatively smooth; a schedule derived from observed metrics improved performance in that case.

Training-loss spikes have been hard to reproduce under controlled conditions, and their cause is not fully understood. Researchers show that simple quadratic models with heavy-tailed, power-law inputs reproduce the spikes, then align per-layer training dynamics of deep networks with such models and give a lightweight simulation algorithm. In a 1.3B-parameter Transformer language model case study, where an AdamW hyperparameter was changed to induce spikes, the simulation from a step-21k checkpoint predicts bumps for the Spiky1 and Spiky2 configurations while Normal stays comparatively smooth; a schedule derived from observed metrics improved performance in that case.