Low-bit inference stops choosing between accuracy and speed: StrataFlux streams refinement bits from RAM and SSD, cutting 2-bit Llama-3.1-8B perplexity from 9.46 to 8.40
Synopsis
StrataFlux, from Cornell and Carnegie Mellon researchers, keeps low-bit weights and KV cache resident on the GPU and fetches only a small set of refinement bits from host RAM or SSD for the computation at hand, recovering accuracy while retaining quantization's memory and speed advantages. On an RTX 5090, 2-bit Llama-3.1-8B-Instruct weights drop WikiText-2 perplexity from 9.46 to 8.40 while keeping 95.4% of the quantized base's throughput; with 2-bit weights and KV, Qwen3-8B's AIME 2024/2025 accuracy rises from 11.0% to 47.1% at unchanged GPU memory use.
Interpretation
The tradeoff between low-bit quantization and offloading higher-precision data is split apart: weights and KV cache stay resident on the GPU in low bits, while extra precision sits in larger, slower host RAM or SSD and is fetched only when needed. Quantization saves memory and bandwidth but can lose accuracy, while offloading preserves precision but can stall generation; the two paths had been treated as mutually exclusive. On an RTX 5090, 2-bit Llama-3.1-8B-Instruct weights lower WikiText-2 perplexity from 9.46 to 8.40 while maintaining 95.4% of the quantized base's throughput.
What arrives is not a full high-precision copy but additional bits that refine the resident low-bit code, so transfer volume is far below moving high-precision data. Weight channels are chosen by input activation magnitude and KV token groups by attention weights computed from resident low-bit keys, so refinement bits are fetched only for a small set of important positions. At 128K context, Llama-3.1-8B-Instruct with 3-bit weights and 2-bit KV reaches perplexity and speed comparable to vLLM's 4-bit AWQ weights with 8-bit KV on an RTX 5090, using about half the GPU memory.
Perspective
Evaluation covers 2–4-bit weights and 2–3-bit KV cache on dedicated GPUs including the RTX 5090, RTX 5070 Ti and RTX PRO 6000, and on unified-memory platforms including DGX Spark and MacBook Air M3, with accuracy measured on WikiText-2, PG-19, RULER, MATH500 and AIME 2024/2025.
StrataFlux is a provisional system name and may change in future revisions; on unified-memory platforms the main results skip cache misses and set the shift to zero because no SSD wait remains for fetch lead and lag to hide; and for weight refinement, channel-wise and neuron-wise MLP schemes show no consistent speed–accuracy winner across models, quantizers, platforms and speed budgets.