MoE experts don't sort by domain label: an SAE splits each feature's activation by expert, separating Java from front-end code
Synopsis
Researchers at ByteDance Seed and Shanghai University of Finance and Economics trained a sparse autoencoder (SAE, a dictionary that reconstructs a layer's output as sparse interpretable features) on the MoE output of layers 4, 30 and 46 of Qwen3-30B-A3B, and used the encoder's affinity in the expert outputs to decompose each feature's activation exactly over the experts. Averaged over a corpus, each expert gets the set of features it principally drives; most of those features have only one or two dominators, and the division of labor runs below coarse domains, down to Java versus front-end web code and geometry versus arithmetic and polynomials.
Interpretation
The unit for studying expert specialization moves from human-assigned domain labels to sparse features the model learns itself, and an expert is described by the set of features it principally drives. Earlier work counted how often an expert was selected on domain-labeled corpora, with the taxonomy fixed in advance, at coarse granularity, and recording selection rather than contribution. On layers 4, 30 and 46 of Qwen3-30B-A3B, every expert dominates a nonempty set of features, and on average 46% to 59% of an expert's set is dominated by that expert alone.
Ordered by contribution, a single feature's activation concentrates on a few experts rather than spreading over the eight the router selects. Gating weights are themselves uneven, so the analysis has to rule out that contribution unevenness is merely a projection of gating unevenness. At layer 30 the first three rank-averaged gating weights are 0.231, 0.169 and 0.137, while the first three contribution values are 0.512, 0.185 and 0.115; at layers 4, 30 and 46 only 2.09, 1.29 and 1.64 experts per (prefix, activated feature) pair are truncated on average, so the ReLU is not the main source of the steepness.
Expert specialization sits below coarse domains such as code or mathematics, and within one coarse domain nearly disjoint experts share the work. Existing analyses asked only whether experts align with preset domains such as code and mathematics, which cannot show divisions inside a domain. At layer 30, 86.7% of the feature explanations in expert 50's dominant set are judged Java development, 78.6% in expert 57 front-end development, 78.1% in expert 1 geometry and 59.1% in expert 76 arithmetic and polynomial; the same themes fall to 1.9% to 6.7% in random samples from the other 127 experts' dominant sets.
Perspective
The analysis covers layers 4, 30 and 46 of Qwen3-30B-A3B, each with 128 experts and 8 activated per token, with one separately trained SAE of 220 features per layer; statistics are computed on a 1.5B-token validation set drawn from the same distribution as the SAE training set with no overlap, and expert labels and domains are produced by GPT-5.5 from feature explanations.
The authors note that the analysis inherits the SAE's reconstruction error and the splitting and absorption of features; Qwen3's pretraining mixture is not disclosed and the statistics come from a corpus of the authors' choosing, so the reading in Section 5.3 that links expert allocation to domain proportions in training data remains a conjecture; results come from a single model, and the authors propose extending the analysis to more and larger MoE models to see which observations are general.