# Spectral Bias

3 related articles

Subscribe to updates

arXiv Nov 7, 2026
Gating each ReLU feature with a partition-of-unity weight stops regression spectral bias from growing with window count, but on Poisson it makes the lowest frequency the slowest mode
In shallow ReLU models, the spectral bias that makes high frequencies learn slowly is tied directly to the condition number of the feature dictionary's Gram matrix. Multiplying each ReLU feature by a hat partition-of-unity weight makes that Gram matrix block-tridiagonal for any local bilinear form, and its block-Toeplitz symbol sets the learning rate of each frequency. Under a regression loss the symbol stays away from zero, and the measured spread of frequency-dependent learning rates falls from four orders of magnitude to within a factor of five at matched model size; under the stiffness matrix the symbol vanishes at zero frequency, so gating leaves conditioning intact and instead makes the lowest frequency the slowest mode.
In shallow ReLU models, the spectral bias that makes high frequencies learn slowly is tied directly to the condition number of the feature dictionary's Gram matrix. Multiplying each ReLU feature by a hat partition-of-unity weight makes that Gram matrix block-tridiagonal for any local bilinear form, and its block-Toeplitz symbol sets the learning rate of each frequency. Under a regression loss the symbol stays away from zero, and the measured spread of frequency-dependent learning rates falls from four orders of magnitude to within a factor of five at matched model size; under the stiffness matrix the symbol vanishes at zero frequency, so gating leaves conditioning intact and instead makes the lowest frequency the slowest mode.
Victorita Dolean（First author, Corresponding author）
Spectral Bias Gated Local ReLU Dictionary Partition of Unity
researchsquare.com Nov 13, 2026
Spectral bias lives in the training kernel, not the wavenumber basis: dense operator learners spread error almost evenly across modes
Training five operator models on the one-dimensional discrete Poisson problem shows that error is not learned wavenumber by wavenumber but is retained along small-eigenvalue directions of the kernel that governs the training update. Sine-aligned spectral and linear Fourier neural operators give each wavenumber its own rate, so error collects at high wavenumbers; dense networks have eigendirections close to random in the sine basis, so error is nearly equal at every wavenumber and relative error grows only because the true solution spectrum decays. Removing the minibatch cross terms leaves the GELU dense network with under 0.1% of squared error below ηλ≈8×10⁻⁵ under either forcing.
Training five operator models on the one-dimensional discrete Poisson problem shows that error is not learned wavenumber by wavenumber but is retained along small-eigenvalue directions of the kernel that governs the training update. Sine-aligned spectral and linear Fourier neural operators give each wavenumber its own rate, so error collects at high wavenumbers; dense networks have eigendirections close to random in the sine basis, so error is nearly equal at every wavenumber and relative error grows only because the true solution spectrum decays. Removing the minibatch cross terms leaves the GELU dense network with under 0.1% of squared error below ηλ≈8×10⁻⁵ under either forcing.
Stanford University University of California, Santa Cruz
Conrad Ainslie（First author）Ashesh Chattopadhyay（Corresponding author）
Spectral Bias Operator Learning Fourier Neural Operator (FNO)Neural Tangent Kernel
arXiv Nov 9, 2026
Guided-wave inspection switches to sparse measurements: PI-SIREN embeds plate equations in a sinusoidal network and finds a back-side milled notch in experiment
Ultrasonic guided-wave inspection of thin-walled structures usually needs repeated finite-element runs or dense scans to obtain a damage-free baseline. PI-SIREN embeds the coupled elastodynamic equations of Mindlin–Reissner plate theory in the training loss of a sinusoidal-activation network, reconstructing the steady-state wavefield in the frequency domain from sparse measurements without temporal discretization. In the numerical ablation the full optimization pipeline reaches a test error of 0.1037, and in experiment the surrogate serves as the baseline that localizes a back-side milled notch from sparse measurements, whereas comparable plain multilayer-perceptron baselines localize no defect at all.
Ultrasonic guided-wave inspection of thin-walled structures usually needs repeated finite-element runs or dense scans to obtain a damage-free baseline. PI-SIREN embeds the coupled elastodynamic equations of Mindlin–Reissner plate theory in the training loss of a sinusoidal-activation network, reconstructing the steady-state wavefield in the frequency domain from sparse measurements without temporal discretization. In the numerical ablation the full optimization pipeline reaches a test error of 0.1037, and in experiment the surrogate serves as the baseline that localizes a back-side milled notch from sparse measurements, whereas comparable plain multilayer-perceptron baselines localize no defect at all.
Lukas Jilke（First author）
PI-SIREN Physics-Informed Neural Network Damage Identification Spectral Bias

You’re all caught up

Gating each ReLU feature with a partition-of-unity weight stops regression spectral bias from growing with window count, but on Poisson it makes the lowest frequency the slowest mode

Spectral bias lives in the training kernel, not the wavenumber basis: dense operator learners spread error almost evenly across modes

Guided-wave inspection switches to sparse measurements: PI-SIREN embeds plate equations in a sinusoidal network and finds a back-side milled notch in experiment