A 70B model on a 4GB card: AirLLM streams weights layer by layer, and 2.8T Kimi K3 fits in 3.72GB

The VRAM barrier for large-model inference drops to the level of a single consumer GPU, without quantizing, distilling, or pruning the model itself. Previously, running a large model on limited VRAM usually meant quantizing or pruning the weights first, shrinking the model before loading it. A 70B model runs on a single 4GB card; Kimi K3 (2.8T) measures 3.72GB end to end on one RTX 6000 Ada, Qwen3.8-Flash-Next (125B) measures 5.95GB on one RTX 4090, and DeepSeek-V3 (671B) runs in about 12GB.

Weights no longer all reside in VRAM at once; they enter the GPU layer by layer for computation and release space once done. Layer-by-layer streaming is the core mechanism, paired with prefetching that overlaps model loading with computation. Kimi K3 uses per-expert streaming, loading only the experts each token actually routes to; Qwen3.8-Flash-Next keeps its n-gram embedding table file-mapped on the host, where a 64GB machine suffices, while decoder layers stream.

The same streaming idea extends to training, with frozen weights flowing through the GPU layer by layer while trainable parameters stay resident as adapters. Previously such low-VRAM schemes targeted inference, while training a large model still required weights and optimizer states to fit in VRAM. Qwen3.8-Flash-Next (125B) trains under 6GB on an RTX 3060 Ti, and Qwen3.8-27B trains in about 2GB at sequence length 512.

The approach targets researchers and engineers who want to run or fine-tune large models on limited VRAM, covering model families from 70B to 2.8T with end-to-end single-card VRAM measurements. Kimi K3 additionally requires compressed-tensors and flash-attn, a CUDA 12 build of torch, and transformers 4.56.x; Qwen3.8-27B requires transformers 5.8+; Qwen3.8-Flash-Next requires a transformers build with qwen4_exp and about 360GB of checkpoint disk space.

The VRAM figures come from the project's own single-card end-to-end measurements rather than independent evaluation, and third-party reproduction has not been reported. Kimi K3's model code mandates flash attention, and no prebuilt flash-attn wheel exists for CUDA 13 yet, so a CUDA 12 build of torch is needed; its remote code does not load on transformers 5.x. Qwen3.8-Flash-Next needs a transformers build with qwen4_exp and about 360GB of checkpoint disk space.