Conrad Ainslie

Training five operator models on the one-dimensional discrete Poisson problem shows that error is not learned wavenumber by wavenumber but is retained along small-eigenvalue directions of the kernel that governs the training update. Sine-aligned spectral and linear Fourier neural operators give each wavenumber its own rate, so error collects at high wavenumbers; dense networks have eigendirections close to random in the sine basis, so error is nearly equal at every wavenumber and relative error grows only because the true solution spectrum decays. Removing the minibatch cross terms leaves the GELU dense network with under 0.1% of squared error below ηλ≈8×10⁻⁵ under either forcing.

Training five operator models on the one-dimensional discrete Poisson problem shows that error is not learned wavenumber by wavenumber but is retained along small-eigenvalue directions of the kernel that governs the training update. Sine-aligned spectral and linear Fourier neural operators give each wavenumber its own rate, so error collects at high wavenumbers; dense networks have eigendirections close to random in the sine basis, so error is nearly equal at every wavenumber and relative error grows only because the true solution spectrum decays. Removing the minibatch cross terms leaves the GELU dense network with under 0.1% of squared error below ηλ≈8×10⁻⁵ under either forcing.