Gaming PCs as one elastic inference platform: FreeToken targets 290B+ MoE models on consumer hardware
Synopsis
FlashML released FreeToken, an edge-native Mixture-of-Experts serving engine aimed at running frontier open-weight MoE models above 290B parameters locally on personal and consumer hardware. It treats GPUs, CPUs, host memory and interconnects as one elastic inference platform, organizing compute and data movement through bandwidth-adaptive CPU–GPU co-execution, full-layer double-buffered prefill streaming and a global LRU expert cache. The engine also adds semantic anchor checkpoints to skip recomputation after agentic context edits and reallocates VRAM at runtime between expert caches and KV memory, covering models such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B and GLM-5.2 across MXFP4, NVFP4, FP8 and BF16 formats.
Interpretation
Frontier-scale open-weight MoE models are now positioned as servable locally on personal and consumer hardware rather than requiring datacenter-class equipment. The release describes GPUs, CPUs, host memory and interconnects as heterogeneous edge resources inside one unified, elastic inference platform rather than as separate parts. The release states the goal of running 290B+ frontier MoE models at interactive speeds on a gaming PC and lists native support for NVIDIA RTX 30, 40 and 50 series GPUs.
The engine splits MoE computation between CPU and GPU according to available bandwidth and streams weights into VRAM to fit consumer memory and bandwidth constraints. This combines a bandwidth-adaptive CPU–GPU co-execution policy, full-layer double-buffered prefill streaming, a global LRU expert cache, graph-compatible execution and the FTW fast weight format. The release lists these components as core runtime features and says they span consumer laptops, gaming desktops and workstation GPUs.
For agent workloads, the engine keeps recurrent state and KV caches in semantic anchor checkpoints so context edits such as tool calls and thinking blocks avoid recomputing the whole context. VRAM can be reallocated at runtime between expert caches and KV memory without restarting the engine or reloading weights. The release describes this capability for real coding and tool-calling agents and offers Anthropic/OpenAI-compatible APIs for integration.
Perspective
The demonstrated scope is the release of an edge-native MoE serving engine for personal and consumer hardware, spanning consumer laptops, gaming desktops and workstation GPUs with native support for NVIDIA RTX 30, 40 and 50 series.
The release describes capabilities as goals and feature lists without measured throughput, latency or memory figures, so interactive speeds for 290B+ models on consumer hardware remain a stated target.