# random walk of signal distance

Deep networks have an edge of chaos in the plane of weight and bias variances where signals and gradients propagate with depth; its location was known only in the infinite-width limit, and whether and how it moves at finite width was disputed. Treating the logarithm of the distance between two inputs as a random walk, this work computes its drift and variance layer by layer and finds the finite-width edge shifted to a larger weight variance, by an amount that falls as width grows. The resulting initialisation rule, Crinit, places a network on that edge without a search and is more accurate and faster to train than standard rules on MNIST and Fashion-MNIST.

Deep networks have an edge of chaos in the plane of weight and bias variances where signals and gradients propagate with depth; its location was known only in the infinite-width limit, and whether and how it moves at finite width was disputed. Treating the logarithm of the distance between two inputs as a random walk, this work computes its drift and variance layer by layer and finds the finite-width edge shifted to a larger weight variance, by an amount that falls as width grows. The resulting initialisation rule, Crinit, places a network on that edge without a search and is more accurate and faster to train than standard rules on MNIST and Fashion-MNIST.