Gyeongmin Kim

LLM quantization usually gives an entire layer one bit width, ignoring that weight blocks within a layer differ in sensitivity to quantization noise. A2BQ scores each 128-row weight block by post-softmax attention variance from an FP16 model on a WikiText-2 calibration set, allocates 4/6/8 bits under a 5.7-bit average budget, and sets qLoRA+ LoRA ranks from layer sensitivity and assigned precision. On TinyLlama-1.1B, the attention-scope configuration with three protected layers at each boundary reaches 7.884 test perplexity at 73.11 tokens/s, while runtime GPU memory stays near FP16 because the reference backend dequantizes weights into floating-point buffers before computing.

LLM quantization usually gives an entire layer one bit width, ignoring that weight blocks within a layer differ in sensitivity to quantization noise. A2BQ scores each 128-row weight block by post-softmax attention variance from an FP16 model on a WikiText-2 calibration set, allocates 4/6/8 bits under a 5.7-bit average budget, and sets qLoRA+ LoRA ranks from layer sensitivity and assigned precision. On TinyLlama-1.1B, the attention-scope configuration with three protected layers at each boundary reaches 7.884 test perplexity at 73.11 tokens/s, while runtime GPU memory stays near FP16 because the reference backend dequantizes weights into floating-point buffers before computing.