Nonlinear attention makes loss decay inversely with depth, while linear attention stays bound to the data spectrum
In controlled in-context learning tasks comparing linear and nonlinear attention, the authors find that nonlinear attention can selectively focus on relevant tokens, letting strong and weak spectral directions be learned in parallel and producing inverse-depth loss decay, whereas linear attention's depth scaling depends on the data spectrum and quickly plateaus.