# KV-cache pruning

The work introduces Universal Attention, which combines query-aware, key-aware, and similarity-based decay mechanisms via a geometric mean into an additive bias on the attention matrix, letting the model adaptively prune its KV-cache while keeping RoPE and Softmax; on a 1B Llama3-style model trained on 50B tokens it achieves over 10x compression, with RULER averages of 50.3/49.3 versus 40.8 for unpruned Llama3 at only 21.4/12.9 MB of cache.

The work introduces Universal Attention, which combines query-aware, key-aware, and similarity-based decay mechanisms via a geometric mean into an additive bias on the attention matrix, letting the model adaptively prune its KV-cache while keeping RoPE and Softmax; on a 1B Llama3-style model trained on 50B tokens it achieves over 10x compression, with RULER averages of 50.3/49.3 versus 40.8 for unpruned Llama3 at only 21.4/12.9 MB of cache.