MoE precision pays twice: store one 4.27 bpw copy, read 2.9 bpw per workload, and stay near 4-bit formats
Synopsis
Mixture-of-experts weight precision carries two separate costs, one for storage and one for decode; DuoRate stores a single 4.27 bpw copy and picks a read prefix per workload from that workload's own prefill, reaching 76.16 average accuracy at a 2.9 bpw read rate on Qwen3-30B-A3B, close to 4-bit formats.
Interpretation
Mixture-of-experts weight precision splits into two separately settable quantities: the stored width that occupies resident memory, and the width a batch-one decode step actually reads. Published per-expert allocators constrain only the first, whether the budget is per block, per layer or global, and routing frequency enters the objective rather than the constraint when it appears at all. The released BitsMoE 2-bit checkpoint stores 2.5425 bpw but a batch-one decode step reads 7.5008 bpw, because the FP16 basis shared by a layer's experts is amortized over all of them in storage and over only the 8 selected at decode.
The read side can therefore be allocated per workload, and without labels. The store side is decided once, the read side once per workload, and the two are no longer pinned to one set of widths. The online stage runs one forward pass over the workload's own prefill, records per-channel second moments and routing counts, scores each expert and tier with one dot product, and meets the read budget by a bisection on a single multiplier in seconds on a CPU.
Routing frequency cancels from every per-expert decision in the decode objective, so it cannot reorder experts. Prior allocators treat routing frequency as an importance signal and weight the distortion term by it. After the Lagrangian relaxation separates across experts, both terms in each summand carry the same positive factor, so the family swept by the multiplier is identical for every positive weight vector and frequency only selects which member meets the budget.
Perspective
The evaluation covers one mixture-of-experts model, Qwen3-30B-A3B, and quantizes only the expert matrices; attention, routers, embeddings and normalization stay in BF16. The efficiency figures are derived from the storage layout, and measuring them end to end requires a prefix-read kernel, which the authors leave to future work. Read tables are calibrated per workload; recalibration is cheap, but a deployment whose request mix drifts must schedule it.
In the measurements prefill also runs at the read depth rather than from the stored tier, so the reported quality is a conservative estimate of the two-stage deployment. The efficiency figures come from the storage layout rather than a fused kernel measurement. Read tables are calibrated per workload, and a deployment whose request mix drifts must schedule recalibration.