Min Chen

Deep networks on modular addition memorize the training set and generalize only long afterward, a delay called grokking. Treating training epoch as temperature and weight decay as a conjugate field, the study builds neuron-to-neuron couplings from second derivatives of the loss in per-neuron norm coordinates, orders neurons along the Fiedler vector of that coupling graph, and runs a real-space renormalization group on the resulting one-dimensional chain. Five critical exponents from the blocking map agree with values fitted directly from 30 independent training runs to within 1%–10%.

Deep networks on modular addition memorize the training set and generalize only long afterward, a delay called grokking. Treating training epoch as temperature and weight decay as a conjugate field, the study builds neuron-to-neuron couplings from second derivatives of the loss in per-neuron norm coordinates, orders neurons along the Fiedler vector of that coupling graph, and runs a real-space renormalization group on the resulting one-dimensional chain. Five critical exponents from the blocking map agree with values fitted directly from 30 independent training runs to within 1%–10%.