# MoE serving

FlashML released FreeToken, an edge-native Mixture-of-Experts serving engine aimed at running frontier open-weight MoE models above 290B parameters locally on personal and consumer hardware. It treats GPUs, CPUs, host memory and interconnects as one elastic inference platform, organizing compute and data movement through bandwidth-adaptive CPU–GPU co-execution, full-layer double-buffered prefill streaming and a global LRU expert cache. The engine also adds semantic anchor checkpoints to skip recomputation after agentic context edits and reallocates VRAM at runtime between expert caches and KV memory, covering models such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B and GLM-5.2 across MXFP4, NVFP4, FP8 and BF16 formats.

FlashML released FreeToken, an edge-native Mixture-of-Experts serving engine aimed at running frontier open-weight MoE models above 290B parameters locally on personal and consumer hardware. It treats GPUs, CPUs, host memory and interconnects as one elastic inference platform, organizing compute and data movement through bandwidth-adaptive CPU–GPU co-execution, full-layer double-buffered prefill streaming and a global LRU expert cache. The engine also adds semantic anchor checkpoints to skip recomputation after agentic context edits and reallocates VRAM at runtime between expert caches and KV memory, covering models such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B and GLM-5.2 across MXFP4, NVFP4, FP8 and BF16 formats.

Mixture-of-experts weight precision carries two separate costs, one for storage and one for decode; DuoRate stores a single 4.27 bpw copy and picks a read prefix per workload from that workload's own prefill, reaching 76.16 average accuracy at a 2.9 bpw read rate on Qwen3-30B-A3B, close to 4-bit formats.

Mixture-of-experts weight precision carries two separate costs, one for storage and one for decode; DuoRate stores a single 4.27 bpw copy and picks a read prefix per workload from that workload's own prefill, reaching 76.16 average accuracy at a 2.9 bpw read rate on Qwen3-30B-A3B, close to 4-bit formats.