Retrieved documents no longer need an identical prefix: one document cache survives changing system prompts, lifting hit rate from 3.3% to 63.9%

KV-cache reuse in LLM serving no longer requires the whole prefix to match character for character; the input is split by delimiter tokens and each segment independently checks whether its cached states can be reused, so a miss in one segment does not block reuse in later ones. Serving engines have mainly optimized exact-prefix matching, so when the same retrieved document appears after a different system prompt, conversation history, or dynamic metadata, its cache cannot be reused and prefill is recomputed. On 2,500 requests sampled from LongBench Pro T5.1 under a Zipfian distribution, with 20% of requests sharing a fixed prefix, KV-Repurpose reaches a 63.9% token-level hit rate at 30 GB capacity versus 3.3% for prefix caching; prefix caching reaches only 17.9% even with 1 TB.

The implementation adds segment-wise cache-aware scheduling to vLLM and applies only a positional correction to reused keys instead of recomputing attention with the preceding prefix. Each segment's keys are stored unrotated and RoPE is applied at their current positions on reuse, which realigns cached keys with their new positions rather than recomputing attention. The system extends vLLM v0.20.0 in fewer than 900 lines of Python, runs GPT-OSS-120B with more than 100 billion parameters on a single NVIDIA H200 GPU with KV cache on NVMe SSD, and uses LMCache as the prefix-caching baseline connector.

The quality cost of reuse splits by task type: single-document question answering and summarization largely hold, while cross-document reasoning drops clearly. Against LMCache prefix caching as the quality-preserving recompute reference, recompute scores 0.42 token-level F1 on QASPER single-document QA and reuse also scores 0.42; on multi-document sets 2WikiMultihopQA falls from 0.80 to 0.50 and MuSiQue from 0.61 to 0.40. Across the 25 LongBench Pro tasks, reuse and recompute average 0.43 and 0.45; restricting multi-document inputs to relevant documents partially recovers performance but leaves a gap, and all scores average three runs.

The testbed targets long-context RAG-style serving and is validated on a single H200 with GPT-OSS-120B and KV cache on NVMe SSD, using English documents from 8K to 128K tokens. It provides a naive segment-reuse baseline; the authors state it is not a state-of-the-art method and was neither compared with nor combined with more sophisticated position-independent caching techniques.

The cross-document reasoning degradation only partially recovers when inputs are restricted to relevant documents, and the authors do not attribute the recovery to either fewer documents or removed irrelevant content. Experiments use a single model, so results may not generalize to other architectures, sizes, or training settings, and large-scale deployment evaluation is left for future work. The authors also report that when reusable caches were generated from prompts containing user information, 35 of 100 names were reproduced after reuse, which is why document caches are now generated from the document alone.