Likelihood tempering in linear Bayesian neural networks makes predictive mean and variance transition at different temperature scales, so no schedule recovers both NNGP posterior moments

For single-hidden-layer linear networks with isotropic Gaussian priors under NTK parameterization, the authors derive the limiting variational predictive distribution under every power-law temperature schedule (Theorem 3.1): the predictive mean and variance undergo phase transitions at different scales, with the mean leaving its prior value below an explicit threshold and the variance departing from the prior only at colder temperatures. Earlier results on temperature scaling with model size (Huix et al., Descours et al., Harvey et al.) cover either the mean-field scaling or random-feature models where the mean is preserved, but not the NTK scaling with both layers trained, where the predictive mean is lost as well. Analytical derivation: the mean threshold balances the KL increase from AM-GM and the triangle inequality against the likelihood decrease from Cauchy-Schwarz (Lemma 3.1), and the covariance limit is characterized by the fixed-point equation (15), with full proofs in Appendix B.3.

There exist temperature schedules that recover the untempered NNGP posterior mean, and schedules that recover its variance, but no power-law schedule recovers both (Corollary 3.1). This gives a precise boundary on what likelihood tempering can remedy for prior dominance in wide networks: the temperature ranges for mean and variance recovery do not overlap. Corollary 3.1 follows directly from the regime-by-regime limits of Theorem 3.1; the mean-recovery and variance-recovery conditions correspond to different temperature exponents and cannot hold simultaneously.

Previously prescribed temperature scalings of the relevant order (such as Harvey et al.'s likelihood upweighting by the ratio of parameter count to sample size) fall in case 4 of Theorem 3.1: their limiting mean is already the least-squares prediction, and their limiting variance exceeds the NNGP posterior variance once the temperature is cold enough. It locates existing temperature prescriptions within the paper's phase-transition diagram, showing that they have already passed the NNGP posterior mean at the mean level. Obtained by comparing case 4 of Theorem 3.1 with the thresholds in Corollary 3.1; this is an analytical conclusion rather than an empirical observation.

A finite-width simulation on synthetic data reproduces the transitions: at the largest width the curves approach their theoretical limits, and at very cold temperatures the posterior variance collapses toward zero, collapsing faster for colder temperatures. It provides finite-width numerical corroboration of the analytical transitions and shows that some variance curves converge slowly. Uses a fixed synthetic dataset with known noise variance, an ICM iteration solving the first-order conditions, and verification that the fits are global minimizers (Appendix F.4); the authors note that variance convergence is slow near the critical temperature.

The results apply to single-hidden-layer linear networks with identity activation, isotropic zero-mean Gaussian priors, a Gaussian likelihood with known noise variance, and the NTK scaling in which sample size and input dimension are fixed while width grows to infinity. Within this setting, it gives concrete guidance for temperature-schedule design: if the predictive mean matters, choose the temperature exponent that recovers the NNGP posterior mean; if the predictive variance matters, choose a colder exponent. The authors note that the multivariate central limit theorem would extend the single-input result to finite marginals of the predictive process, and that for general design matrices the isotropic second-moment condition can be enforced by a whitening transformation (Remark A.1).

The results concern global minimizers of the variational objective; whether gradient-based optimization reaches them at large width, and whether it exhibits the same phase transitions, is explicitly left as a separate question. Extension to nonlinear or deeper networks, and to larger variational families that allow dependence between each neuron's input and output weights, remains to be studied. The authors also note that variational approximations of cold posteriors, where likelihood and prior are jointly rescaled, are not yet analyzed, and at which temperature scaling the different predictive moments could be recovered there is an open question. In addition, the simulation shows slow convergence of the variance curves near the critical temperature, so empirical judgments near the threshold at finite width warrant care.