# PoissonComp

Long-context inference makes the KV cache grow linearly with sequence length and often exceed GPU memory. VerKV keeps the full cache in CPU memory and only a Poisson sample of tokens, drawn from importance scores, on the GPU; at each decode step a certificate estimates the attention error and extra entries are fetched from CPU only when the bound exceeds tolerance. Across Qwen3-4B, Qwen3.5-9B and Ministral-3-8B on RULER and LOFT at 32K and 128K context, with 3.13% of the cache resident on GPU, the three retain 94.6%, 84.6% and 79.6% of dense quality, where Fast KVzip and KVzip at the same 3.13% retain only 1.9% to 11.9%.

Long-context inference makes the KV cache grow linearly with sequence length and often exceed GPU memory. VerKV keeps the full cache in CPU memory and only a Poisson sample of tokens, drawn from importance scores, on the GPU; at each decode step a certificate estimates the attention error and extra entries are fetched from CPU only when the bound exceeds tolerance. Across Qwen3-4B, Qwen3.5-9B and Ministral-3-8B on RULER and LOFT at 32K and 128K context, with 3.13% of the cache resident on GPU, the three retain 94.6%, 84.6% and 79.6% of dense quality, where Fast KVzip and KVzip at the same 3.13% retain only 1.9% to 11.9%.