Frontier MoE models on a consumer PC no longer lean on the SSD: Tideway pools VRAM and RAM for a 1.97x average end-to-end speedup

When a frontier-scale MoE model runs on a consumer PC, its weights are forced to spread across VRAM, host RAM and SSD; Tideway schedules the first two as one unified memory pool rather than as independent caches. Earlier systems treated VRAM as a cache of host RAM and host RAM as a cache of the SSD, replacing each tier independently; FreeToken assumes the full expert pool resides in host RAM, llama.cpp delegates paging to OS memory-mapped files, and SSD-LLaMA keeps one layout across phases. On an RTX 5090 with 32 GB of VRAM and host-RAM budgets of 16–128 GiB, across 270 fixed-shape configurations of three open-weight MoE models, Tideway achieves a 1.97x average and up to 3.18x end-to-end speedup over the fastest baseline.

Inside the pool, the layout switches by phase: prefill reserves space for weight streaming and activations and anchors reusable host-RAM experts, while decode reclaims that space as fixed-address expert slots. During prefill each 8K-token chunk activates nearly every expert, and a layer-by-layer scan evicts experts used by the previous layer; Tideway anchors reusable experts and leaves only a small streaming window to absorb SSD reads. In a DeepSeek-V4-Flash setting with 64 GiB host RAM, 32K input and 8K prefill chunks, the SSD supplies 86–99% of the baselines' prefill expert traffic, whereas Tideway fetches only 52% from the SSD.

The execution engine keeps CUDA Graph replay alive under expert paging by splitting each layer into reusable piecewise graphs, with dynamic host paging performed between segments. Default offloading engines abandon graph capture along the paged path and fall back to eager kernel dispatch, adding host dispatch and synchronization overhead. In a profile with 32K input, 1K output and 64 GiB host RAM, CPU orchestration overhead and GPU stalls dominate decode time in most baselines, while llama.cpp spends 92% of decode time computing experts on the CPU.

Evaluation runs on a single consumer desktop: a 32-core AMD CPU, an RTX 5090 with 32 GB of VRAM, PCIe 5.0 and an 8 TB NVMe SSD, with host RAM capped at 16–128 GiB per model, covering three open-weight MoE models and replays of eight TraceLab coding-agent sessions.

Tideway reserves KV-cache capacity before allocating expert slots, so growing contexts leave less memory for expert residency; managing KV blocks inside the pool is listed as a future direction rather than a completed evaluation.