Attention linearization stops treating all dimensions alike: grouping the few outliers cuts Qwen2.5-0.5B perplexity from 31.71 to 27.50
Approximation error in pretrained attention concentrates in a few high-contribution query–key dimensions, and sharing quadratic-kernel coefficients by contribution group lowers frozen-attention relative MSE to 0.432 on Qwen2.5-0.5B and WikiText-103 perplexity from 31.71 to 27.50 after replacing middle-layer attention.