Neurons have no positions, yet they can be coarse-grained: a renormalization-group map predicts five grokking exponents within 1%–10% of direct fits
Synopsis
Deep networks on modular addition memorize the training set and generalize only long afterward, a delay called grokking. Treating training epoch as temperature and weight decay as a conjugate field, the study builds neuron-to-neuron couplings from second derivatives of the loss in per-neuron norm coordinates, orders neurons along the Fiedler vector of that coupling graph, and runs a real-space renormalization group on the resulting one-dimensional chain. Five critical exponents from the blocking map agree with values fitted directly from 30 independent training runs to within 1%–10%.
Interpretation
Hidden neurons are placed on a coarse-grainable one-dimensional chain, letting the delayed generalization of grokking be described quantitatively in the language of real-space renormalization. Earlier phase-transition accounts of grokking relied on isolated observables, and links between renormalization and deep learning had remained largely conceptual, without a coarse-graining derived from the network's internal representation. On a two-layer quadratic-activation network trained on modular addition, coupling strength decays monotonically with distance under Fiedler ordering, the nearest neuron pairs carry stronger mean coupling than the farthest, and a random permutation gives a flat profile.
Training is translated into thermodynamic quantities: the total squared weight norm is the order parameter, weight decay is its conjugate field, and the training epoch acts as an effective inverse temperature. Temperature, field, order parameter and susceptibility have no ready counterparts in a training run, and the statistical ensemble is formed by independent training seeds rather than a Gibbs distribution. All 30 independent runs reach test accuracy between epochs 170 and 182; cross-seed variances of the norm and of the loss serve as proxies for susceptibility and heat capacity, aligned at the susceptibility peak and the maximum test-accuracy slope.
Merging adjacent neurons into block spins compresses the coupling matrix to two coordinates, whose linearized map yields two relevant scaling directions. The network supplies neither distances between neurons nor a lattice on which blocks could form, and the coupling-space map has no closed analytical form. A weighted least-squares fit over a 65-configuration parameter cloud around the grokking center gives thermal and magnetic eigenvalues both above one, from which five critical exponents follow.
The renormalization-group predictions for five critical exponents agree with direct measurements within 1%–10%, and finite-size scaling across seven moduli gives about 3% deviation. Direct values come from separate fits, so the Rushbrooke, Josephson and Widom scaling identities can be tested rather than satisfied by construction. The correlation-length exponent fitted from transition widths across seven moduli is 1.400 against a renormalization-group value of 1.444, and rescaling the seven transition curves by the renormalization-group exponent collapses them onto a common curve.
Perspective
The demonstrated setting is a bias-free two-layer perceptron with quadratic activation on modular addition, full-batch mean-squared-error loss and AdamW training, with 30 independent runs reaching test accuracy between epochs 170 and 182; all thermodynamic quantities are effective descriptions, and the ensemble is a seed ensemble rather than a Gibbs ensemble.
Trained models are not strictly equilibrium samples, and AdamW's decoupled weight decay does not literally reduce to an explicit term added to the loss; the negative thermal eigenvalue is attributed to off-diagonal dominance of the truncated map, so a two-step map with a positive eigenvalue is used instead; finite-size scaling rests on two hypotheses, that different moduli are controlled by the same effective scaling regime and that the task-coordinate circumference sets the characteristic linear size.