The edge of chaos shifts at finite width: Crinit places deep networks on it without a search and trains them faster

The edge of chaos of a finite-width network is no longer the line given by the infinite-width formula; it moves toward a larger weight variance, and the shift falls as width grows. Infinite-width mean-field theory gave a closed-form edge of chaos, while at finite width analytic expansions and simulations had reached conflicting conclusions about whether and in which direction the location moves. For Gaussian weights, simulations of the exact finite-width recursion locate the sign change of the measured drift in agreement with the first-order formula within statistical error; at smaller widths the measured shift at fixed bias exceeds the first-order formula, by a fraction that falls as width grows.

The logarithm of the distance between two inputs performs a random walk through the layers, and its drift decides whether the distance grows or shrinks in a typical network. Mean-field theory replaces each layer's average over neurons by its expectation, but at finite width that average fluctuates from layer to layer, and the effect of those fluctuations on the distance between two inputs had not been computed in full. Along the infinite-width edge of chaos, the drift formula agrees with simulation to within one to two per cent and the variance to within a few per cent, up to eleven per cent at the smallest width.

Backpropagated gradients follow the same random walk, so the same finite-width edge keeps the typical gradient from vanishing or exploding. Gradient vanishing and explosion are usually treated as a problem separate from signal propagation, whereas here both are set by the same drift condition. The median squared gradient some layers below the output is far below its value at the output on the infinite-width edge, and stays at the same order on the finite-width edge; the real cross-entropy gradient on MNIST images follows the same walk within errors.

Along the edge of chaos the information about input similarity stays constant until compression exceeds about one nat, which fixes where to sit on the line and yields the search-free initialisation rule Crinit. The edge of chaos fixes only one combination of weight and bias variances and leaves the steady signal size free, and no measurement had given a quantitative basis for choosing it. On MNIST at width 512, Crinit reaches accuracies of 0.9517 and 0.903 at depths 32 and 64, against 0.9392 and 0.567 for the standard He rule and 0.9438 and 0.713 for LeCun's rule; at depth 64 Crinit reaches training accuracy in 49791 steps against 50488 for the mean-field edge of chaos.

The results concern fully connected networks at initialisation, with Gaussian or orthogonal weights and tanh-like or ReLU-like activations; the training tests use the two small data sets MNIST and Fashion-MNIST, two widths and two depths, and no normalisation layers or skip connections. The expansion requires the signal to settle within far fewer than n layers, so that the layer-to-layer fluctuations do not accumulate before the signal settles.

The first-order shift formula fails at the end of the zero-bias line, where the point must be found by solving without linearising; with orthogonal weights the measured shift is about four times smaller than theory, of opposite sign to the Gaussian case, and the authors report not having traced the discrepancy. The training runs mainly test the operating point and the first layer, while the correction leaves final accuracy unchanged within error and shows in training speed. In deep, narrow networks the zero-bias point compresses more than the budget allows, and Crinit cannot lower the compression with a non-negative bias variance.