AirLLM runs the 2.8T Kimi K3 on one GPU at 3.72GB of VRAM by streaming layers, not quantizing
Synopsis
AirLLM pushes large-model inference memory down to a few gigabytes on a single card: Kimi K3 (2.8T) measured end to end at 3.72GB of VRAM on one RTX 6000 Ada, Qwen3.8-Flash-Next (125B) at 5.95GB on one RTX 4090, and DeepSeek-V3 (671B) at about 12GB. It uses no quantization, distillation or pruning; instead it streams weights layer by layer, and for MoE models loads only the experts each token actually routes to. The project extends the same idea to training: frozen weights stream in one layer at a time while adapters stay on the GPU, letting a 125B model train under 6GB.
Interpretation
The memory floor for large-model inference drops to a few gigabytes on one card: the 2.8T-parameter Kimi K3 measured end to end at 3.72GB of VRAM on a single RTX 6000 Ada, and the 125B Qwen3.8-Flash-Next at 5.95GB on a single RTX 4090. AirLLM explicitly avoids the three weight-shrinking routes of quantization, distillation and pruning, keeping weights off the card and bringing them in on demand so that a 70B-class model runs on a single 4GB GPU card. These figures are end-to-end VRAM measurements for each model on the named single card, covering three separate configurations: K3, Qwen3.8-Flash-Next, and Qwen3.8-27B at 3.33GB on an RTX 3090.
The method splits the model by layer and streams only the current layer's weights into VRAM during inference, swapping in the next layer once it is done. For mixture-of-experts models, per-expert streaming moves only the experts each token actually routes to, rather than every expert in the layer. Kimi K3's 3.72GB was measured on a single RTX 6000 Ada under this per-expert streaming scheme; Qwen3.8-Flash-Next's roughly 51B n-gram embedding table is file-mapped on the host, where a 64GB machine suffices, while decoder layers stream as usual.
The same streaming idea carries over to training: frozen weights flow through one layer at a time while trainable adapters stay resident on the GPU. Training support is a newly added capability, so the 125B Qwen3.8-Flash-Next can train on a small-VRAM card rather than only run inference there. The 125B Qwen3.8-Flash-Next trains under 6GB of VRAM on an RTX 3060 Ti, and Qwen3.8-27B trains in about 2GB at sequence length 512.
Perspective
The demonstrated scope is end-to-end VRAM use for specific models on named single cards, including K3 at 3.72GB on an RTX 6000 Ada, Qwen3.8-Flash-Next at 5.95GB on an RTX 4090 and Qwen3.8-27B at 3.33GB on an RTX 3090; on the training side, a 125B model trains under 6GB on an RTX 3060 Ti.
Kimi K3 carries extra environment requirements: its model code mandates flash attention, it needs a CUDA 12 build of torch because no prebuilt flash-attn wheel exists for CUDA 13 yet, and transformers must be 4.56.x since its remote code does not load on 5.x; Qwen3.8-Flash-Next needs a transformers build with qwen4_exp and about 360GB of checkpoint disk.