Low-bit inference stops choosing between accuracy and speed: StrataFlux streams refinement bits from RAM and SSD, cutting 2-bit Llama-3.1-8B perplexity from 9.46 to 8.40
StrataFlux, from Cornell and Carnegie Mellon researchers, keeps low-bit weights and KV cache resident on the GPU and fetches only a small set of refinement bits from host RAM or SSD for the computation at hand, recovering accuracy while retaining quantization's memory and speed advantages. On an RTX 5090, 2-bit Llama-3.1-8B-Instruct weights drop WikiText-2 perplexity from 9.46 to 8.40 while keeping 95.4% of the quantized base's throughput; with 2-bit weights and KV, Qwen3-8B's AIME 2024/2025 accuracy rises from 11.0% to 47.1% at unchanged GPU memory use.