Expert shape breaks free from routing granularity: near-square experts plus layerwise capacity allocation top nine downstream tasks at 8B
Latent mixture-of-experts decouples expert input dimension from residual width, making expert shape a tunable design choice; at 3.4B and 8B scales, near-square experts with capacity allocated by layerwise intrinsic dimension reach lower validation loss than matched uniform configurations.