Tianzhe Chu

Mixture-of-experts transformers apply conditional computation only to the MLP, so every token still passes through all attention parameters. NESTOR extends the same routing idea to attention: each token is sent to a few columns, each column has its own attention and experts, and the two levels of sparsity nest hierarchically. In language-model pretraining from 100M to 1B active parameters, the MoE transformer needs 1.13–1.29 times the compute and the dense transformer 3.1–4.1 times the compute to reach the same held-out loss, and this factor shows no downward trend as scale grows.

Mixture-of-experts transformers apply conditional computation only to the MLP, so every token still passes through all attention parameters. NESTOR extends the same routing idea to attention: each token is sent to a few columns, each column has its own attention and experts, and the two levels of sparsity nest hierarchically. In language-model pretraining from 100M to 1B active parameters, the MoE transformer needs 1.13–1.29 times the compute and the dense transformer 3.1–4.1 times the compute to reach the same held-out loss, and this factor shows no downward trend as scale grows.