# Mixture-of-Experts

Sakana AI and Modal put Sakana Namazu, a roughly 1-trillion-parameter mixture-of-experts model, into production API service at NVFP4 precision. Compression was applied only to the routed experts that hold most of the parameters, while attention, the router and the shared experts stayed at BF16, and calibration used Japanese-heavy conversations so the model matched its production checkpoint on Japanese instruction following. A custom draft model trained on more than 30 billion raw tokens raised speculative token acceptance by 25% over a generic drafter; a second drafter retrained on about 1 million real requests lifted average acceptance length from 3.51 to 5.01 and cut p99 latency from 68s to 52s across 1,000 held-out requests.

Sakana AI and Modal put Sakana Namazu, a roughly 1-trillion-parameter mixture-of-experts model, into production API service at NVFP4 precision. Compression was applied only to the routed experts that hold most of the parameters, while attention, the router and the shared experts stayed at BF16, and calibration used Japanese-heavy conversations so the model matched its production checkpoint on Japanese instruction following. A custom draft model trained on more than 30 billion raw tokens raised speculative token acceptance by 25% over a generic drafter; a second drafter retrained on about 1 million real requests lifted average acceptance length from 3.51 to 5.01 and cut p99 latency from 68s to 52s across 1,000 held-out requests.

Recursive self-improvement had only been closed on post-training, where each trial costs GPU-hours and returns a verifiable score; a frontier pre-training run costs hundreds of thousands of GPU-hours, cannot be repeated, and has no single-score objective. A newly formed team at Shanghai Innovation Institute and Xinxu AI split pre-training into smaller closable loops, using agents for debugging, tiny-SFT and related measures to estimate post-training headroom, and quantified workflows for data tasks, producing the 30B-A3B base model 801 in three months. On 13 base benchmarks it averages 73.13, behind only the larger Qwen3.5-35B-A3B at 76.67.

Recursive self-improvement had only been closed on post-training, where each trial costs GPU-hours and returns a verifiable score; a frontier pre-training run costs hundreds of thousands of GPU-hours, cannot be repeated, and has no single-score objective. A newly formed team at Shanghai Innovation Institute and Xinxu AI split pre-training into smaller closable loops, using agents for debugging, tiny-SFT and related measures to estimate post-training headroom, and quantified workflows for data tasks, producing the 30B-A3B base model 801 in three months. On 13 base benchmarks it averages 73.13, behind only the larger Qwen3.5-35B-A3B at 76.67.

Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.

Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.

Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.

Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.

Prior cost models estimated speculative decoding from the expert-weight bytes activated during verification, predicting a concave speedup curve with limited gains at low batch size for sparse MoE targets. Across 10 sparse MoE models from 30B to 1T parameters and 8 to 512 experts, fine-grained models instead often peak near batch size 1. Two effects explain it: routing overlap across adjacent draft positions cuts the unique experts loaded during verification by 21% to 34% relative to an independence baseline, and GPU execution dynamics, namely bandwidth-utilization gaps between MoE and attention kernels plus L2 reuse, decouple runtime from expert bytes.

Prior cost models estimated speculative decoding from the expert-weight bytes activated during verification, predicting a concave speedup curve with limited gains at low batch size for sparse MoE targets. Across 10 sparse MoE models from 30B to 1T parameters and 8 to 512 experts, fine-grained models instead often peak near batch size 1. Two effects explain it: routing overlap across adjacent draft positions cuts the unique experts loaded during verification by 21% to 34% relative to an independence baseline, and GPU execution dynamics, namely bandwidth-utilization gaps between MoE and attention kernels plus L2 reuse, decouple runtime from expert bytes.

Researchers at ByteDance Seed and Shanghai University of Finance and Economics trained a sparse autoencoder (SAE, a dictionary that reconstructs a layer's output as sparse interpretable features) on the MoE output of layers 4, 30 and 46 of Qwen3-30B-A3B, and used the encoder's affinity in the expert outputs to decompose each feature's activation exactly over the experts. Averaged over a corpus, each expert gets the set of features it principally drives; most of those features have only one or two dominators, and the division of labor runs below coarse domains, down to Java versus front-end web code and geometry versus arithmetic and polynomials.

Researchers at ByteDance Seed and Shanghai University of Finance and Economics trained a sparse autoencoder (SAE, a dictionary that reconstructs a layer's output as sparse interpretable features) on the MoE output of layers 4, 30 and 46 of Qwen3-30B-A3B, and used the encoder's affinity in the expert outputs to decompose each feature's activation exactly over the experts. Averaged over a corpus, each expert gets the set of features it principally drives; most of those features have only one or two dominators, and the division of labor runs below coarse domains, down to Java versus front-end web code and geometry versus arithmetic and polynomials.

When a user queries a large model, nothing in the reply shows it came from the model the developer claims, so a routing service swapping in a cheaper model or an attacker tampering with the token stream goes unnoticed. Anthropic's open-source FlashProve attaches a zero-knowledge succinct argument (zkSNARK, a short certificate a verifier can check quickly without the weights) to inference. On 4-bit Llama 4 Maverick, a 400B-parameter mixture of experts, a single H100 generates proofs at about 110 tokens per second, roughly 19 MB per 1,024-token request, at about 70 to 95 times the GPU time vLLM needs to run the model itself.

When a user queries a large model, nothing in the reply shows it came from the model the developer claims, so a routing service swapping in a cheaper model or an attacker tampering with the token stream goes unnoticed. Anthropic's open-source FlashProve attaches a zero-knowledge succinct argument (zkSNARK, a short certificate a verifier can check quickly without the weights) to inference. On 4-bit Llama 4 Maverick, a 400B-parameter mixture of experts, a single H100 generates proofs at about 110 tokens per second, roughly 19 MB per 1,024-token request, at about 70 to 95 times the GPU time vLLM needs to run the model itself.

Mistral released Mistral Large 4 (Le Chonk) on Hugging Face, a multimodal Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context window, shipped as open weights in FP8 and NVFP4 variants. It pairs a large expert pool with sparse attention and a Pixtral vision encoder, so the language model selects a few experts per token while an attention indexer selects which earlier positions to attend to. On the Artificial Analysis Cyber Index test that asks a model to reproduce a real vulnerability in open-source software and then patch it, the model scores 82%, which Mistral describes as the highest reported score on that test.

Mistral released Mistral Large 4 (Le Chonk) on Hugging Face, a multimodal Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context window, shipped as open weights in FP8 and NVFP4 variants. It pairs a large expert pool with sparse attention and a Pixtral vision encoder, so the language model selects a few experts per token while an attention indexer selects which earlier positions to attend to. On the Artificial Analysis Cyber Index test that asks a model to reproduce a real vulnerability in open-source software and then patch it, the model scores 82%, which Mistral describes as the highest reported score on that test.