Pulin Gong

Deep-learning training outcomes are hard to reproduce, usually blamed on random seeds, batch order and other external noise. This work treats training as a dynamical system in parameter space and proposes that the basins leading to different solutions interleave arbitrarily closely, forming a riddled-basin geometry. In a minimal two-layer model this yields a near-zero uncertainty exponent: initialization precision must improve by almost 24 orders of magnitude to halve the fraction of disagreeing outcomes, and across roughly 500,000 deep-network trainings the analogous parameter-space and function-space exponents are also near zero.

Deep-learning training outcomes are hard to reproduce, usually blamed on random seeds, batch order and other external noise. This work treats training as a dynamical system in parameter space and proposes that the basins leading to different solutions interleave arbitrarily closely, forming a riddled-basin geometry. In a minimal two-layer model this yields a near-zero uncertainty exponent: initialization precision must improve by almost 24 orders of magnitude to halve the fraction of disagreeing outcomes, and across roughly 500,000 deep-network trainings the analogous parameter-space and function-space exponents are also near zero.