# FP8

3 related articles

Subscribe to updates

lyogavin/airllm Oct 12, 2026
AirLLM runs the 2.8T Kimi K3 on one GPU at 3.72GB of VRAM by streaming layers, not quantizing
AirLLM pushes large-model inference memory down to a few gigabytes on a single card: Kimi K3 (2.8T) measured end to end at 3.72GB of VRAM on one RTX 6000 Ada, Qwen3.8-Flash-Next (125B) at 5.95GB on one RTX 4090, and DeepSeek-V3 (671B) at about 12GB. It uses no quantization, distillation or pruning; instead it streams weights layer by layer, and for MoE models loads only the experts each token actually routes to. The project extends the same idea to training: frozen weights stream in one layer at a time while adapters stay on the GPU, letting a 125B model train under 6GB.
AirLLM pushes large-model inference memory down to a few gigabytes on a single card: Kimi K3 (2.8T) measured end to end at 3.72GB of VRAM on one RTX 6000 Ada, Qwen3.8-Flash-Next (125B) at 5.95GB on one RTX 4090, and DeepSeek-V3 (671B) at about 12GB. It uses no quantization, distillation or pruning; instead it streams weights layer by layer, and for MoE models loads only the experts each token actually routes to. The project extends the same idea to training: frozen weights stream in one layer at a time while adapters stay on the GPU, letting a 125B model train under 6GB.
AirLLM Large language models FP8
sarvam.ai Nov 3, 2026
Sarvam's Prism decision model answers in 44 ms and averages 71.8 across three indices, the best average under 100 ms among 25 models tested
Sarvam released Prism, a decision model built on Sarvam 105B for enterprise workflows that need fast, calibrated judgments. Across 25 decision models Sarvam tested, Prism averaged 71.8 over three indices with a 44 ms median response, the highest average among models answering under 100 ms; higher-scoring models took roughly two to six times as long. The weights are open so enterprises can train on their own data and run on their own infrastructure.
Sarvam released Prism, a decision model built on Sarvam 105B for enterprise workflows that need fast, calibrated judgments. Across 25 decision models Sarvam tested, Prism averaged 71.8 over three indices with a 44 ms median response, the highest average among models answering under 100 ms; higher-scoring models took roughly two to six times as long. The weights are open so enterprises can train on their own data and run on their own infrastructure.
Sarvam Prism System One decision model Fine-tuning FP8
Hugging Face - Blog Oct 20, 2026
PyTorch 2.15 on the Mac: MoE decoding 5–6.5× faster and video VAE memory down from 270 GiB to 8 GiB
PyTorch 2.15 brings FP8 tensors, a native grouped matmul and a batch of dedicated Metal kernels to the Mac's MPS backend, making local LLM decoding and video generation faster and lighter on memory. Previously many MPS operators went through MPSGraph, MoE decoding copied group offsets to the CPU and looped over groups, and tiled video VAE decoding pushed driver memory to 270 GiB. On an M3 Ultra, Qwen3.6-35B-A3B decoding rises from 9.4 to 61.3 tokens/s, and Wan2.2 VAE decoding peaks near 8 GiB.
PyTorch 2.15 brings FP8 tensors, a native grouped matmul and a batch of dedicated Metal kernels to the Mac's MPS backend, making local LLM decoding and video generation faster and lighter on memory. Previously many MPS operators went through MPSGraph, MoE decoding copied group offsets to the CPU and looped over groups, and tiled video VAE decoding pushed driver memory to 270 GiB. On an M3 Ultra, Qwen3.6-35B-A3B decoding rises from 9.4 to 61.3 tokens/s, and Wan2.2 VAE decoding peaks near 8 GiB.
Hugging Face
PyTorch 2.15 MPS backend Metal kernels Grouped matmul FP8

You’re all caught up

AirLLM runs the 2.8T Kimi K3 on one GPU at 3.72GB of VRAM by streaming layers, not quantizing

Sarvam's Prism decision model answers in 44 ms and averages 71.8 across three indices, the best average under 100 ms among 25 models tested

PyTorch 2.15 on the Mac: MoE decoding 5–6.5× faster and video VAE memory down from 270 GiB to 8 GiB