Sakana Namazu goes live in NVFP4: the trillion-parameter model keeps its Japanese, and a custom drafter cuts p99 latency from 68s to 52s
Synopsis
Sakana AI and Modal put Sakana Namazu, a roughly 1-trillion-parameter mixture-of-experts model, into production API service at NVFP4 precision. Compression was applied only to the routed experts that hold most of the parameters, while attention, the router and the shared experts stayed at BF16, and calibration used Japanese-heavy conversations so the model matched its production checkpoint on Japanese instruction following. A custom draft model trained on more than 30 billion raw tokens raised speculative token acceptance by 25% over a generic drafter; a second drafter retrained on about 1 million real requests lifted average acceptance length from 3.51 to 5.01 and cut p99 latency from 68s to 52s across 1,000 held-out requests.
Interpretation
Sakana Namazu, a model of roughly 1 trillion parameters, now serves API inference at NVFP4 on Modal, a lower-precision format built for NVIDIA Blackwell GPUs that cuts memory and raises throughput. At full precision the model's memory footprint requires very large clusters, and standard post-training quantization recipes shrink that footprint but can degrade model quality, especially for non-English models, because the usual calibration datasets do not capture language register or tool-calling behavior. The quantized model matched Sakana's production checkpoint on Japanese instruction following, within the noise floor of Sakana's own published benchmarks, while running at NVFP4 speed and memory.
The quantization scheme compresses only the routed experts, which hold the bulk of the parameters, while attention, the router and the shared experts stay at BF16. Rather than applying an off-the-shelf recipe, the team first measured how Sakana Namazu's weights responded to compression under NVFP4 numerics and built the approach around what it found. Calibration ran on conversations in Sakana Namazu's style, weighted heavily toward Japanese, so all 23,040 experts saw representative traffic before their scales were fixed.
A custom DFlash draft model was trained on the exact token IDs Sakana Namazu produced, not on text that had been decoded and re-tokenized. Generic draft models are trained on generic text and guess Sakana Namazu's Japanese business register poorly, yielding acceptance rates under 40%. The training corpus exceeded 30 billion tokens, and the drafter raised speculative token acceptance by 25% over a generic draft model, dropped per-token decode latency by about 35%, and let each replica carry about 40% more traffic before queue latencies climb.
A second drafter, trained from scratch after launch, moved to the DFlash2 architecture, which holds up better when customers sample at nonzero temperatures, covering most of Sakana Namazu's traffic. The first drafter was trained before Sakana Namazu had served a single public request, so it learned from representative traffic rather than the real thing. Two identical copies of the serving setup ran side by side on the same hardware and replayed 1,000 held-out requests with the sampling settings customers actually send; average acceptance length rose from 3.51 to 5.01, a 43% gain, and p99 latency fell from 68s to 52s.
Perspective
The demonstrated scope is Sakana Namazu's production deployment on Modal's GPU pool at NVFP4, running tensor-parallel inference across four GPUs, together with the quantization, drafter and routing improvements measured on that deployment; the figures belong to this model and this serving configuration.
The Japanese instruction-following comparison is described as within the noise floor of Sakana's own published benchmarks rather than an exact match, and the first drafter learned from representative traffic while only the second used roughly 1 million real requests.