# Memory-efficient LLM inference

AirLLM cuts the VRAM needed for large-model inference to what a single consumer GPU can hold: instead of quantization, distillation, or pruning, it streams weights layer by layer, so a 70B model runs on a 4GB card. Its 2026 updates report end-to-end measurements of 3.72GB for Kimi K3 (2.8T) on one RTX 6000 Ada, 5.95GB for Qwen3.8-Flash-Next (125B) on one RTX 4090, and about 12GB for DeepSeek-V3 (671B). The same mechanism extends to training, where frozen weights stream through while adapters stay on the GPU, letting a 125B model train under 6GB.

AirLLM cuts the VRAM needed for large-model inference to what a single consumer GPU can hold: instead of quantization, distillation, or pruning, it streams weights layer by layer, so a 70B model runs on a 4GB card. Its 2026 updates report end-to-end measurements of 3.72GB for Kimi K3 (2.8T) on one RTX 6000 Ada, 5.95GB for Qwen3.8-Flash-Next (125B) on one RTX 4090, and about 12GB for DeepSeek-V3 (671B). The same mechanism extends to training, where frozen weights stream through while adapters stay on the GPU, letting a 125B model train under 6GB.