MLP linearization moves to low-degree rational fits: Qwen3-8B drops 14% of parameters and decodes faster
Synopsis
Instead of relying only on polynomial random features, MLP linearization now approximates activations with low-degree rational functions and splits each layer, via partial fractions, into separable weight-side and input-side inner products. Across synthetic fitting, vision, 3D and language models this route gives lower error at the same feature budget; compressing Qwen3-8B by replacing its last eight feed-forward layers removes 1.17B parameters (14% of the model) and raises decoding throughput from 763.3 to 809.6 tokens/s, with reasoning and knowledge benchmarks roughly at parity with the uncompressed model.
Interpretation
Feed-forward linearization can now be built on low-degree rational approximations rather than only polynomial or Fourier random features. Earlier linearization routes were either limited by the Fourier transform of the activation, as with unbounded functions such as ReLU, or used polynomial approximation, which diverges as the activation argument grows. In synthetic experiments comparing the fully connected layer output with the linearized prediction by mean squared error at a shared feature budget, the rational linearization achieves lower error across all tested activations.
The construction writes the activation as partial fractions and builds separable features for each piece. Partial fractions split a rational function into a polynomial part plus irreducible quadratic terms, whose denominators have no real root and correspond to conjugate pole pairs. Each irreducible quadratic is first discretized by Gauss-Laguerre nodes for the frequency integral, then estimated per frequency with Performers-style softmax-kernel random features for the cosine and sine components, separating weights and inputs into an inner product of two feature maps.
Error and variance of the rational block are controlled by different quantities, so adding poles lowers approximation error without raising variance. For the polynomial block, reaching higher accuracy makes the relative variance grow exponentially with degree, so the cost rises quickly. The analysis bounds total error by three terms, the rational fit error, the quadrature truncation error and the finite projection count, with the rational block's variance set by the input scale alone and independent of the number of poles.
Perspective
The demonstrated scope covers synthetic feed-forward fitting, ViT classification and transfer on CIFAR-100, Places365, ImageNet-1k and VTAB-1k, PTv3 3D semantic segmentation, NeRF and SIREN coordinate networks, a physics-informed network for the Burgers equation, GPT-2 Large pretraining and Qwen3-8B compression. For Qwen3-8B the last four or last eight feed-forward layers are replaced, with supervised fine-tuning and on-policy distillation recovering accuracy. It suits researchers and engineers working on feed-forward efficiency and inference deployment.
The rational block's variance depends on the input scale, and a poorly chosen scale inflates the feature exponent, so per-layer calibration is needed. Under VTAB-1k, where each task provides only 1000 training examples, approximation error is harder to compensate with so little data. When compressing Qwen3-8B, pass@16 on long-chain benchmarks such as AIME remains close to the dense model, indicating the underlying capability is largely preserved; the error compounds with generation length, which the distillation stage mainly mitigates.