Same document, different system prompt, full recompute: segment-wise reuse lifts cache hit rate from 3.3% to 63.9%

Key-value cache reuse in large language model serving no longer requires the whole prompt to match verbatim. The system splits input at delimiter tokens into segments, checks the cache for each segment independently, skips prefill on a hit, and computes and stores states only on a miss. Serving engines have mainly optimized exact-prefix matching, so when the same document follows a different system prompt, conversation history, or dynamic metadata, its cache cannot be reused and the states are recomputed. On 2,500 requests sampled from a Zipfian RAG-style workload where 20% of requests share a fixed prefix, segment-wise reuse reaches a 63.9% hit rate at 30 GB of cache versus 3.3% for prefix caching; prefix caching reaches only 17.9% even with 1 TB of cache.

On reuse, the system keeps each stored segment's keys unrotated and reapplies rotary position encoding according to their current positions, realigning cached keys without recomputing attention with the preceding prefix. Prior position-independent caching work mainly targets preserving output quality under positional mismatch, whereas this testbed targets how many more reuse opportunities appear when exact prefix matching fails. The testbed extends vLLM v0.20.0, serves GPT-OSS-120B on a single NVIDIA H200 with the cache on NVMe SSD, adds under 900 lines of Python, and evaluates quality on LongBench Pro plus seven LongBench datasets with scores averaged over three runs.

The testbed targets researchers and serving engineers developing and evaluating segment-wise key-value cache reuse on vLLM, covering cache generation, prompt construction, eviction, and runtime monitoring, and it preserves output quality on single-document question answering and summarization. The authors leave evaluation of large-scale deployments to future work and name comparing or combining segment-wise reuse with existing position-independent caching techniques as an important direction.

The authors state that the implementation applies only a simple positional correction and neither combines nor directly compares with more sophisticated position-independent caching techniques, and that experiments use a single large model, so results may not generalize to other architectures, sizes, or training settings. Naive reuse still degrades clearly on cross-document reasoning, and filtering to relevant documents with reference answers only partially recovers it. The appendix capacity extrapolation relies on assumptions such as 15 KB of cache per byte of document and a 20% reuse probability, which the authors say are only for interpreting scale and do not characterize a specific production workload.