Attention now picks its own parameters per token: NESTOR nests expert routing inside column routing and lowers pretraining loss at equal compute
Synopsis
Mixture-of-experts transformers apply conditional computation only to the MLP, so every token still passes through all attention parameters. NESTOR extends the same routing idea to attention: each token is sent to a few columns, each column has its own attention and experts, and the two levels of sparsity nest hierarchically. In language-model pretraining from 100M to 1B active parameters, the MoE transformer needs 1.13–1.29 times the compute and the dense transformer 3.1–4.1 times the compute to reach the same held-out loss, and this factor shows no downward trend as scale grows.
Interpretation
Conditional computation no longer happens only in the MLP; the attention operator itself selects parameters and context per token. The earlier MoE transformer replaced each MLP with many experts and activated only a few per token, but every token in a layer still shared the same attention parameters. At the four active-parameter scales of 100M, 200M, 500M and 1B, NESTOR has lower held-out loss than both the iso-sparsity MoE and the dense baseline; extrapolating the fitted loss-compute curves, matching NESTOR's loss at the largest scale is estimated to need about 1.22 times the compute for the MoE and about 4.1 times for the dense model.
A column router first sends each token to a few columns, each column carries its own attention and its own experts, and an expert router then selects within the column. Column routing is computed once at the entrance of a block of M layers and held fixed across those layers, so one decision fixes which attention parameters are used, which tokens are attended to, and which experts follow. The experiments use C=8 columns with the number of active columns rising with depth from 3 in the first block to 8 in the last; models are trained on DCLM-baseline at context length 4096 for 50 tokens per active parameter and compared with MoE and dense baselines that use full causal attention.
Perspective
The demonstrated scope is language-model pretraining from 100M to 1B active parameters, continual pretraining on four target domains, and context extension from 4k to 64k, all evaluated by held-out cross-entropy; the authors propose testing next whether the compute multiplier persists at larger scales and carries over to downstream tasks such as standard benchmarks after post-training.
The authors note that the largest model has 1B active parameters and that all models are evaluated by held-out loss; the theoretical analysis concerns a simplified one-layer model with linear attention and squared loss, which the authors say reproduces the pretraining compute efficiency, the growing advantage with context length, and the reduced forgetting, but under that stylized setting.