FOSCAR stops dequantizing each KV value: folding scales and offsets into the matmul lets 2-bit decode attention beat BF16 on H100 and B200
Synopsis
Low-bit KV caches cut memory traffic, yet existing kernels still convert every integer back to floating point and apply its scale and offset, so instruction issue becomes the new bottleneck. FOSCAR folds each quantization group's shared scale and offset into the other matrix operand or the partial product, and builds Tensor Core operands directly from packed integers with bit manipulation. Its INT2 kernels reach geometric-mean speedups over FlashInfer BF16 on H100 and B200, and the kernel library is open-sourced.
Interpretation
When decode attention streams a low-bit KV cache, the bottleneck shifts from memory bandwidth to instruction issue: reconstructing each integer as floating point and applying its group scale and offset consumes scarce issue slots. Earlier kernels such as KIVI, QServe, BitDecoding, SAW-INT4 and OSCAR convert or reconstruct low-bit values before every matrix product, so per-element preparation limits latency gains even when dequantized K and V are never written back to HBM. On H100, reconstructing four BF16 Tensor Core operands from INT4 takes six instructions, or six lane-slots per warp lane, while four INT4 values occupy only two bytes; on B200 the same reconstruction takes twelve instructions. Measured as issue demand per byte against the hardware peak, INT2 reconstruction exceeds B200's peak, and INT4 reconstruction alone consumes a substantial share of that budget.
FOSCAR does not apply scale and offset per value; it multiplies each group's shared scale into the other matrix input or the partial product and adds the offset as an output correction. This is called affine folding and splits by quantization scheme: token-wise K uses accumulator folding on the QK partial product, channel-wise K scales the query first, and V folds correspondingly into the PV input or partial product. In exact arithmetic, affine folding yields the same attention output as dequantizing every value before the matrix products; ablations against per-value reconstruction show an added speedup on H100 and B200, and switching the folded B200 INT2 kernel from FP8 to FP4 KV operands adds a further gain.
Perspective
The demonstrated scope is decode-attention kernels on NVIDIA H100 and B200, plus serving throughput measured with FOSCAR integrated into SGLang, serving Qwen3-8B with 100K input and 1K output tokens at 99% or 95% prefix-cache hit rates; quantization groups are token-wise for K and V with shared FP32 scales and offsets.
In the serving accuracy comparison, every question-paired 95% confidence interval for the accuracy difference on GPQA-Diamond and GSM8K spans zero, so five runs show no measurable gain or loss but do not establish formal equivalence; INT2 kernel error is in the same range as OSCAR INT2, indicating the error comes from 2-bit quantization rather than affine folding.