ResidualQuant anchors on the final-loop KV and stores 2-bit residuals, cutting looped-Transformer KV storage by 80.7% at near-BF16 accuracy under INT2/4
To address the KV cache memory bottleneck that grows with recurrent depth in looped Transformers, the authors propose ResidualQuant, which uses the final-loop KV states as an anchor and represents the remaining loops as low-precision residuals, combined with least-squares scaling, rotations applied to the residuals, and loop-wise INT4/INT2 mixed precision; across Ouro-1.4B and Huginn-3.5B on mathematical reasoning and code generation benchmarks it improves the accuracy-memory tradeoff over rotation-based KV quantization, reducing theoretical KV storage by 80.7% relative to BF16 while retaining accuracy close to BF16 under mixed precision.