Spectral bias lives in the training kernel, not the wavenumber basis: dense operator learners spread error almost evenly across modes
Synopsis
Training five operator models on the one-dimensional discrete Poisson problem shows that error is not learned wavenumber by wavenumber but is retained along small-eigenvalue directions of the kernel that governs the training update. Sine-aligned spectral and linear Fourier neural operators give each wavenumber its own rate, so error collects at high wavenumbers; dense networks have eigendirections close to random in the sine basis, so error is nearly equal at every wavenumber and relative error grows only because the true solution spectrum decays. Removing the minibatch cross terms leaves the GELU dense network with under 0.1% of squared error below ηλ≈8×10⁻⁵ under either forcing.
Interpretation
Spectral bias is relocated: it is a property of the eigenbasis of the kernel that governs the training update, not of the wavenumber basis in which it is usually reported. Earlier explanations assumed each mode is learned at a rate decreasing with wavenumber, and in the neural tangent kernel picture training converges fastest along the kernel eigenfunctions with the largest eigenvalues. Across smooth and white forcing, the ideal diagonal spectral operator, the linear and nonlinear Fourier neural operators, the GELU dense network and the deep linear dense network all follow the same rule: a model that does not learn the operator retains its error in the kernel's small-eigenvalue directions.
The kernel's small eigenvalues fix how much error remains, and the projection of those slow eigendirections onto the sine modes fixes the error's shape in wavenumber. Slow rates can come from the data spectrum, from the network nonlinearity through coupling between training samples, or from truncation in the representation itself, each a different source of retained error. For the GELU dense network under either forcing, the cumulative error mass of the full stacked kernel reaches the independently measured RMS error, while replacing the kernel with its block-diagonal part leaves under 0.1% of squared error below ηλ≈8×10⁻⁵.
The slow eigendirections of the dense networks are indistinguishable from random directions in the sine basis, so every wavenumber receives the same share of error and forms a flat floor. Sine-aligned models give each wavenumber its own rate and concentrate error at high wavenumbers, whereas a flat floor makes relative error grow with wavenumber. Under smooth forcing the final RMS errors of the linear Fourier neural operator, the nonlinear Fourier neural operator, the GELU dense network and the deep linear dense network are 3.6×10⁻⁵, 3.3×10⁻⁵, 4.2×10⁻⁵ and 2.6×10⁻⁴, with spectra ranging from rising and jagged to flat.
Perspective
The conclusions rest on the inverse of the one-dimensional discrete Poisson operator with N=128 interior grid points, five fixed architectures, and smooth and white forcing ensembles, with kernel statistics averaged over the last 100 training steps.
The kernel moves during training, so the error decomposition holds only over a window in which it is nearly fixed; the authors also note that the structure of the stacked kernel is hard to predict, so whether a given architecture will show spectral bias cannot be determined before training.