# Adam Optimizer

With architecture and data held fixed, a checkpoint-only reconstruction attack recovers training images best from Muon-trained models and worst from Adam-trained models, making the optimizer choice a control axis for training-data privacy.

With architecture and data held fixed, a checkpoint-only reconstruction attack recovers training images best from Muon-trained models and worst from Adam-trained models, making the optimizer choice a control axis for training-data privacy.

Training-loss spikes have been hard to reproduce under controlled conditions, and their cause is not fully understood. Researchers show that simple quadratic models with heavy-tailed, power-law inputs reproduce the spikes, then align per-layer training dynamics of deep networks with such models and give a lightweight simulation algorithm. In a 1.3B-parameter Transformer language model case study, where an AdamW hyperparameter was changed to induce spikes, the simulation from a step-21k checkpoint predicts bumps for the Spiky1 and Spiky2 configurations while Normal stays comparatively smooth; a schedule derived from observed metrics improved performance in that case.

Training-loss spikes have been hard to reproduce under controlled conditions, and their cause is not fully understood. Researchers show that simple quadratic models with heavy-tailed, power-law inputs reproduce the spikes, then align per-layer training dynamics of deep networks with such models and give a lightweight simulation algorithm. In a 1.3B-parameter Transformer language model case study, where an AdamW hyperparameter was changed to induce spikes, the simulation from a step-21k checkpoint predicts bumps for the Spiky1 and Spiky2 configurations while Normal stays comparatively smooth; a schedule derived from observed metrics improved performance in that case.