# Grokking

Deep networks on modular addition memorize the training set and generalize only long afterward, a delay called grokking. Treating training epoch as temperature and weight decay as a conjugate field, the study builds neuron-to-neuron couplings from second derivatives of the loss in per-neuron norm coordinates, orders neurons along the Fiedler vector of that coupling graph, and runs a real-space renormalization group on the resulting one-dimensional chain. Five critical exponents from the blocking map agree with values fitted directly from 30 independent training runs to within 1%–10%.

Deep networks on modular addition memorize the training set and generalize only long afterward, a delay called grokking. Treating training epoch as temperature and weight decay as a conjugate field, the study builds neuron-to-neuron couplings from second derivatives of the loss in per-neuron norm coordinates, orders neurons along the Fiedler vector of that coupling graph, and runs a real-space renormalization group on the resulting one-dimensional chain. Five critical exponents from the blocking map agree with values fitted directly from 30 independent training runs to within 1%–10%.

A two-layer, 427k-parameter transformer shows grokking on the finite-field discrete logarithm map: after a long memorization phase it reaches 99.8% held-out accuracy at p = 113. A 12-model capacity sweep from 90k to 14.4M parameters finds grokking recurring at multiple scales, with a non-monotonic region whose outcome changes when training data increase; hidden states grow aligned with multiplicative-character structure, and prime-order and disjoint-generator controls also generalize.

A two-layer, 427k-parameter transformer shows grokking on the finite-field discrete logarithm map: after a long memorization phase it reaches 99.8% held-out accuracy at p = 113. A 12-model capacity sweep from 90k to 14.4M parameters finds grokking recurring at multiple scales, with a non-monotonic region whose outcome changes when training data increase; hidden states grow aligned with multiplicative-character structure, and prime-order and disjoint-generator controls also generalize.