# KV Cache Reuse

7 related articles

Subscribe to updates

arXiv Nov 4, 2026
Where a KV-cache repair budget should go: recomputing the answer-bearing chunk beats token selection at 5%
Non-prefix KV-cache reuse encodes retrieved chunks separately and concatenates their caches, losing 19% to 100% of context-dependent questions that full prefill answers, with same-kind distractor chunks adding to the loss. Comparing whether a small repair budget should go to distractors or to the answer-bearing chunk, the study finds that restoring all distractors costs nearly a full prefill, while recomputing the answer-bearing chunk beats token-level selectors at a 5% budget in all nine open-domain QA and API-documentation settings.
Non-prefix KV-cache reuse encodes retrieved chunks separately and concatenates their caches, losing 19% to 100% of context-dependent questions that full prefill answers, with same-kind distractor chunks adding to the loss. Comparing whether a small repair budget should go to distractors or to the answer-bearing chunk, the study finds that restoring all distractors costs nearly a full prefill, while recomputing the answer-bearing chunk beats token-level selectors at a 5% budget in all nine open-domain QA and API-documentation settings.
Beijing Normal University Beijing Normal University, Zhuhai Beijing Normal-Hong Kong Baptist University
Yuxing Peng（First author）
KV Cache Reuse Contextual Compatibility KV-Cache Repair Budget Allocation
arXiv Nov 6, 2026
Nebula lifts position-independent caching to the cluster: median time-to-first-token falls 61.4% on average across four models and four workloads
For retrieval-augmented generation and agent workloads, Nebula promotes position-independent caching (PIC) segments, which let the same text segment reuse already-computed key-value cache under a different prefix, into cluster-level cache objects, enabling remote reuse and parallel prefill. Distributed caching systems had reused key-value cache only under strict prefix matching, while single-node PIC could not draw on other nodes' cache or compute. Across four models and four real RAG and agentic workloads, Nebula cuts median time-to-first-token by an average of 61.4% against the strongest baseline and recovers an average of 76.1% of the tokens missed by prefix caching.
For retrieval-augmented generation and agent workloads, Nebula promotes position-independent caching (PIC) segments, which let the same text segment reuse already-computed key-value cache under a different prefix, into cluster-level cache objects, enabling remote reuse and parallel prefill. Distributed caching systems had reused key-value cache only under strict prefix matching, while single-node PIC could not draw on other nodes' cache or compute. Across four models and four real RAG and agentic workloads, Nebula cuts median time-to-first-token by an average of 61.4% against the strongest baseline and recovers an average of 76.1% of the tokens missed by prefix caching.
Yifei Liu（First author）
Nebula distributed LLM serving framework Position-Independent Caching KV Cache Reuse Prefix Caching
KVCache.AI Nov 8, 2026
AMD's ATOM inference engine gains a shared Mooncake KV pool, roughly doubling throughput over split pools on four MI350X servers
Coding agents resend the same long conversation, so whether the server still holds that history outside GPU memory decides how many tokens each turn must recompute. AMD's open-source ATOM inference engine, which places prefill and decode on separate GPUs, now connects to Mooncake Store to merge both halves of host memory into one KV pool that prefill saves to and restores from. On four AMD Instinct MI350X servers, the shared pool reached about twice the throughput of per-server pools at 96 and 112 concurrent sessions, with KV reuse held at 94–96%.
Coding agents resend the same long conversation, so whether the server still holds that history outside GPU memory decides how many tokens each turn must recompute. AMD's open-source ATOM inference engine, which places prefill and decode on separate GPUs, now connects to Mooncake Store to merge both halves of host memory into one KV pool that prefill saves to and restores from. On four AMD Instinct MI350X servers, the shared pool reached about twice the throughput of per-server pools at 96 and 112 concurrent sessions, with KV reuse held at 94–96%.
Mooncake Store KV Cache Reuse AgentX
arXiv Oct 12, 2026
Retrieved documents no longer need an identical prefix: one document cache survives changing system prompts, lifting hit rate from 3.3% to 63.9%
KV-Repurpose changes KV-cache reuse in large language model serving from requiring an identical prompt prefix to deciding reuse per segment, so a retrieved document can reuse its cached states even when system prompts, conversation histories, or timestamps differ. Under a Zipfian RAG workload at 30 GB cache capacity, the token-level hit rate rises from 3.3% for prefix caching to 63.9%, lowering time to first token; single-document question answering and summarization largely hold, while cross-document reasoning degrades.
KV-Repurpose changes KV-cache reuse in large language model serving from requiring an identical prompt prefix to deciding reuse per segment, so a retrieved document can reuse its cached states even when system prompts, conversation histories, or timestamps differ. Under a Zipfian RAG workload at 30 GB cache capacity, the token-level hit rate rises from 3.3% for prefix caching to 63.9%, lowering time to first token; single-document question answering and summarization largely hold, while cross-document reasoning degrades.
KIOXIA Corporation (Japan)
Taiga Ikeda（First author）
KV-Repurpose KV Cache Reuse KV Cache Retrieval-Augmented Generation
arXiv Oct 2, 2026
HeteroFold lets Llama, Qwen, and Ministral share KV caches across families, running about 10.7x faster than native prefill at 32K context
The work proposes HeteroFold, a prefill-free cross-family KV cache transfer method that keeps both sender and receiver frozen and, through token alignment over shared character boundaries, cross-layer and cross-head K/V mapping, and receiver-aware calibration, achieves the best cache-transfer performance on all four long-context benchmarks across six transfer directions among Llama-3.1-8B, Qwen3-4B, and Ministral-3-14B, matches text-based communication on the HiddenBench multi-agent benchmark, and at 32K context makes Llama-3.1-8B to Ministral-3-14B transfer about 10.7x faster than native prefill.
The work proposes HeteroFold, a prefill-free cross-family KV cache transfer method that keeps both sender and receiver frozen and, through token alignment over shared character boundaries, cross-layer and cross-head K/V mapping, and receiver-aware calibration, achieves the best cache-transfer performance on all four long-context benchmarks across six transfer directions among Llama-3.1-8B, Qwen3-4B, and Ministral-3-14B, matches text-based communication on the HiddenBench multi-agent benchmark, and at 32K context makes Llama-3.1-8B to Ministral-3-14B transfer about 10.7x faster than native prefill.
New York University University of Southern California Seoul National University
Vincent-Daniel Yun（First author）
HeteroFold KV Cache Transfer KV Cache Reuse Multi-Agent LLM Systems
arXiv Sep 29, 2026
PReCache shares KV caches across multi-LoRA agents via low-rank precomputation and neutral reconstruction, cutting TTFT by up to several times with almost no accuracy loss
The work presents PReCache, a training-free KV cache sharing framework with two designs: PreLRShared precomputes a compact agent-specific low-rank cache for every agent when each segment of the shared context is first processed, removing later agents' repeated prefill over the accumulated trajectory, while ReBaseShared further reconstructs the shared base cache from adapter-free hidden states to reduce dependence on the previous agent, with lazy prefill for single-stream inference and double batching for concurrent serving; across LLaMA-3.
The work presents PReCache, a training-free KV cache sharing framework with two designs: PreLRShared precomputes a compact agent-specific low-rank cache for every agent when each segment of the shared context is first processed, removing later agents' repeated prefill over the accumulated trajectory, while ReBaseShared further reconstructs the shared base cache from adapter-free hidden states to reduce dependence on the previous agent, with lazy prefill for single-stream inference and double batching for concurrent serving; across LLaMA-3.
Seoul National University
Hyesung Jeon（First author）
PReCache KV Cache Reuse LoRA Adapters LLM Agents
arXiv Sep 29, 2026
KVCMAS shares multi-agent KV caches via low-rank anchor pools and chained correction, delivering a 2.0 TTFT speedup over non-shared inference at 32K shared tokens and 8 QPS while cutting peak GPU memory to about 1/3.7 of KVComm
The work presents KVCMAS, an online KV cache correction framework for prompt-specialized multi-agent systems that represents cross-agent cache deviations as compact low-rank states and chains corrections along the agent workflow without an additional context-free reference prefill, matching or improving the accuracy of prior KV cache sharing methods across language and vision-language workloads while achieving a 2.0 TTFT speedup over inference without KV cache sharing at 32K shared tokens and 8 QPS and reducing peak GPU memory by up to 3.7 relative to the prior correction method KVComm.
The work presents KVCMAS, an online KV cache correction framework for prompt-specialized multi-agent systems that represents cross-agent cache deviations as compact low-rank states and chains corrections along the agent workflow without an additional context-free reference prefill, matching or improving the accuracy of prior KV cache sharing methods across language and vision-language workloads while achieving a 2.0 TTFT speedup over inference without KV cache sharing at 32K shared tokens and 8 QPS and reducing peak GPU memory by up to 3.7 relative to the prior correction method KVComm.
Seoul National University
Hyesung Jeon（First author）
KVCMAS KV cache correction KV Cache Reuse multi-agent systems

You’re all caught up

Where a KV-cache repair budget should go: recomputing the answer-bearing chunk beats token selection at 5%

Nebula lifts position-independent caching to the cluster: median time-to-first-token falls 61.4% on average across four models and four workloads

AMD's ATOM inference engine gains a shared Mooncake KV pool, roughly doubling throughput over split pools on four MI350X servers

Retrieved documents no longer need an identical prefix: one document cache survives changing system prompts, lifting hit rate from 3.3% to 63.9%

HeteroFold lets Llama, Qwen, and Ministral share KV caches across families, running about 10.7x faster than native prefill at 32K context

PReCache shares KV caches across multi-LoRA agents via low-rank precomputation and neutral reconstruction, cutting TTFT by up to several times with almost no accuracy loss

KVCMAS shares multi-agent KV caches via low-rank anchor pools and chained correction, delivering a 2.0 TTFT speedup over non-shared inference at 32K shared tokens and 8 QPS while cutting peak GPU memory to about 1/3.7 of KVComm