Donglin Zhuang

Low-bit KV caches cut memory traffic, yet existing kernels still convert every integer back to floating point and apply its scale and offset, so instruction issue becomes the new bottleneck. FOSCAR folds each quantization group's shared scale and offset into the other matrix operand or the partial product, and builds Tensor Core operands directly from packed integers with bit manipulation. Its INT2 kernels reach geometric-mean speedups over FlashInfer BF16 on H100 and B200, and the kernel library is open-sourced.

Low-bit KV caches cut memory traffic, yet existing kernels still convert every integer back to floating point and apply its scale and offset, so instruction issue becomes the new bottleneck. FOSCAR folds each quantization group's shared scale and offset into the other matrix operand or the partial product, and builds Tensor Core operands directly from packed integers with bit manipulation. Its INT2 kernels reach geometric-mean speedups over FlashInfer BF16 on H100 and B200, and the kernel library is open-sourced.