AdamW's fixed points match an implicit objective, but its own dynamics can leave them unstable

AdamW's fixed points can be written as the stationary points of a static implicit objective built from a parameter constraint and a barrier-like regularizer that varies with the parameters, so a converging AdamW can only stop at such a stationary point. Earlier readings treated AdamW's decoupled weight decay as a dynamic penalty that depends on optimizer state rather than a static objective depending only on parameters, and AdamW had no general convergence proof. The equivalence between the fixed-point condition and the stationarity condition of the implicit objective is proved in both directions, and the regularizer diverges as parameters approach the constraint boundary, so the constraint holds naturally.

Matching fixed points does not mean stability: a stable equilibrium of AdamW must be a strict local minimum of the implicit objective, but a local minimum also needs an extra Hessian bound, so some strict local minima of the implicit objective are unstable under AdamW. For gradient flow, strict local minima are automatically asymptotically stable and nondegenerate saddles automatically unstable; AdamW is not a gradient flow of the implicit objective, so stability is set by the Jacobian of its own vector field and must be checked separately, which fails once weight decay is nonzero. In one dimension the Routh–Hurwitz criterion on a cubic characteristic polynomial gives exact conditions; when the weight-decay and momentum relation turns the condition into an upper bound, a quadratic objective violates it at the stated parameters and its discrete fixed point is unstable too; in the common momentum regime a non-convex quartic violates the lower bound and the trajectory oscillates persistently.

Multiplying weight decay by a factor tied to the strength of the gradient update gives AdamMe; that factor equals one at every equilibrium, so the fixed points and implicit objective are unchanged while the continuous-time dynamics gain a Lyapunov function. AdamW's instability comes from a mismatch in its update, where the first-moment estimate is normalized by the second-moment estimate but the weight decay on the parameters is not, rather than from the implicit objective itself. Under the stated conditions AdamMe's Lyapunov function is nonincreasing along trajectories and every bounded trajectory approaches the equilibrium set; on GPT-2-Small (124M), with hyperparameters tuned for each method, AdamMe closely tracks AdamW and reaches a slightly lower validation loss.

The analysis works in the continuous-time limit with exact (full-batch) gradients and infinitesimal step size, and stability refers to equilibria of that ordinary differential equation; discrete-time AdamW is treated separately, with exact criteria in one dimension and only sufficient conditions in higher dimensions. The empirical comparison is one GPT-2-Small (124M) training run, with learning rate and weight decay tuned for each method.

Higher dimensions give only sufficient conditions, and exact stability still requires the spectrum of the full Jacobian. How much this non-convergence matters for large-scale neural network training remains unclear: networks need not converge to be useful, and in the experiments training appears robust to the instability. The discrete-time stability conditions for AdamW are complicated even in one dimension and are therefore reported separately.