Last-layer weights alone can rank model quality: a statistical account of heavy-tailed spectra, with an overfitting marker during training
Lead
A Bayes free energy computed from trained final-layer weights alone tracks model quality across convolutional networks and LLMs, and under a heavy-tailed spectral model it yields an optimal heaviness whose crossing marks the onset of overfitting.
Story
Model quality can now be read off trained weight matrices alone, without training or test data. Earlier model-quality metrics are data-dependent, expensive on massive training sets, and impossible to compute when training data are not released. Across ImageNet-1K pretrained torchvision models the free energy shows strong cross-group Spearman correlation with top-5 test accuracy, and it correlates strongly with LAMBADA accuracy for Pythia models and with MMLU-Pro for Qwen3.5.
The construction splits a network into nonlinear features and final-layer weights, then takes the marginal likelihood over features. Under the interpolation assumption the minimum-norm solution carries all information linking features to targets, so the marginal likelihood can be written using only final-layer weights. The derivation runs inside an entropic regularization framework at a temperature that concentrates the marginal likelihood on the interpolator submanifold, leaving a relative free energy with a trace term and a log-determinant term.
Assuming weight spectra follow the High-Temperature Marchenko-Pastur distribution makes the free energy convex in a scale-based heaviness parameter with a unique optimum. The link between heavy-tailed spectra and model quality had been an empirical observation without a statistical justification. Taking the deterministic limit of the free energy under the double-asymptotic scaling of random matrix theory yields an optimal ratio, and for ResNet50 trained on CIFAR-10 and CIFAR-100 the minimum test error coincides with that ratio.
What to watch
The free energy is derived only for final-layer weights and for regression, and it is evaluated on models trained to interpolation. For convolutional networks and LLMs it follows test error during training; for convolutional networks the minimum test error occurs where the heavy-tail ratio reaches its optimum. The authors propose that deriving that ratio a priori would allow monitoring during training and early stopping once it is crossed.
The authors note that the free energy is derived only for final-layer weights and for regression, and that the dual formulation cannot immediately be compared across models with different activation dimension. For convolutional models during training, the prior precision parameter must be tuned post-hoc to obtain good correlation. Models using dropout before the final layer and vision transformers did not show correlation; the authors conjecture that dropout prevents complex correlations in final-layer weights, and that the feature-and-weight decomposition may be too crude for vision transformers.