# AirLLM

2 related articles

Subscribe to updates

lyogavin/airllm Oct 12, 2026
AirLLM runs the 2.8T Kimi K3 on one GPU at 3.72GB of VRAM by streaming layers, not quantizing
AirLLM pushes large-model inference memory down to a few gigabytes on a single card: Kimi K3 (2.8T) measured end to end at 3.72GB of VRAM on one RTX 6000 Ada, Qwen3.8-Flash-Next (125B) at 5.95GB on one RTX 4090, and DeepSeek-V3 (671B) at about 12GB. It uses no quantization, distillation or pruning; instead it streams weights layer by layer, and for MoE models loads only the experts each token actually routes to. The project extends the same idea to training: frozen weights stream in one layer at a time while adapters stay on the GPU, letting a 125B model train under 6GB.
AirLLM pushes large-model inference memory down to a few gigabytes on a single card: Kimi K3 (2.8T) measured end to end at 3.72GB of VRAM on one RTX 6000 Ada, Qwen3.8-Flash-Next (125B) at 5.95GB on one RTX 4090, and DeepSeek-V3 (671B) at about 12GB. It uses no quantization, distillation or pruning; instead it streams weights layer by layer, and for MoE models loads only the experts each token actually routes to. The project extends the same idea to training: frozen weights stream in one layer at a time while adapters stay on the GPU, letting a 125B model train under 6GB.
AirLLM Large language models FP8
lyogavin/airllm Oct 13, 2026
A 70B model on a 4GB card: AirLLM streams weights layer by layer, and 2.8T Kimi K3 fits in 3.72GB
AirLLM cuts the VRAM needed for large-model inference to what a single consumer GPU can hold: instead of quantization, distillation, or pruning, it streams weights layer by layer, so a 70B model runs on a 4GB card. Its 2026 updates report end-to-end measurements of 3.72GB for Kimi K3 (2.8T) on one RTX 6000 Ada, 5.95GB for Qwen3.8-Flash-Next (125B) on one RTX 4090, and about 12GB for DeepSeek-V3 (671B). The same mechanism extends to training, where frozen weights stream through while adapters stay on the GPU, letting a 125B model train under 6GB.
AirLLM cuts the VRAM needed for large-model inference to what a single consumer GPU can hold: instead of quantization, distillation, or pruning, it streams weights layer by layer, so a 70B model runs on a 4GB card. Its 2026 updates report end-to-end measurements of 3.72GB for Kimi K3 (2.8T) on one RTX 6000 Ada, 5.95GB for Qwen3.8-Flash-Next (125B) on one RTX 4090, and about 12GB for DeepSeek-V3 (671B). The same mechanism extends to training, where frozen weights stream through while adapters stay on the GPU, letting a 125B model train under 6GB.
AirLLM Memory-efficient LLM inference Large language models

You’re all caught up

AirLLM runs the 2.8T Kimi K3 on one GPU at 3.72GB of VRAM by streaming layers, not quantizing

A 70B model on a 4GB card: AirLLM streams weights layer by layer, and 2.8T Kimi K3 fits in 3.72GB