Why deep-learning runs resist reproduction: interwoven basins leave outcomes unpredictable no matter how precise the initialization

Unpredictability of training outcomes is traced to a geometric structure: initializations leading to different solutions interleave arbitrarily closely in parameter space, so every initialization has a nearby starting point that ends elsewhere. Earlier work mostly attributed irreproducibility to external random sources such as random initialization, mini-batch ordering, data augmentation and GPU numerical error, leaving unexplained how a one-bit perturbation can produce variability comparable to several noise sources combined. In a minimal two-layer network at learning rate 2.5, the chaotic attractor in the permutation-invariant plane has longitudinal Lyapunov exponents of 0.1564 and 0.0256 and transverse exponents of −0.0645 and −0.2047, satisfying the theoretical conditions for a riddled basin.

This geometry imposes a precision-independent floor on predictability: raising initialization precision cannot reliably predict which solution training will reach. Previously reported trainability boundaries were zero-measure skinny fractals, where predictability is recovered once initial-condition uncertainty is made small enough; here the fractal has positive measure and the boundary is inseparable from the basin. The minimal model gives an uncertainty exponent of 0.0126±0.0002, meaning initial uncertainty must shrink by almost 24 orders of magnitude to halve the fraction of differing outcomes; on deep networks including VGG-12, the parameter-space exponent fitted on both CPU and GPU is 0.000±0.002.

Irreproducibility extends beyond parameters to the input-output function a network learns, so representing the same function with different parameters does not remove the difference. Parameter-space irreproducibility does not automatically imply function-space irreproducibility, because symmetries let different parameter configurations represent the same function, and linear networks even converge to initialization-independent solutions. In the representative VGG-12 setting the functional uncertainty exponent is 0.00±0.01, and churn, the fraction of inputs on which two networks predict differently, plateaus at about 2.9% as initialization precision increases; results agree across 32 functional similarity measures from the literature.

The conclusions rest on analytic and numerical analysis of a minimal two-layer model and on compute-constrained scaled configurations of VGG-12, ResNet and BERT; the representative deep-network experiments use MNIST with 50% label noise and fix mini-batch order with CPU computation to ensure determinism. The functional uncertainty exponent is positioned as a practical, architecture-agnostic diagnostic for whether a training configuration sits in a riddling-like regime.

The authors call riddled basin geometry a candidate organizing principle for irreproducibility rather than an established universal mechanism; in the minimal model they observe a sequence of phase transitions with increasing learning rate, from no riddling through pseudo-riddling and true riddling to transient riddling. Symmetry removal can mitigate initialization dependence, but the resulting basin geometry remains to be characterized because destinations are no longer identifiable through matched or vanishing neurons. Why performance and irreproducibility often coincide near the optimal learning rate, and whether an analytic link to generalization bounds exists, remain open questions.