# Prefix Caching

4 related articles

Subscribe to updates

arXiv Nov 10, 2026
Model switches no longer throw away the prefix cache: ShadowPrefill warms the unchosen model in idle capacity, cutting switch-call latency by up to 21.4%
When a per-step router sends an agent's next call to a different model, the target must re-prefill the session context, and the prefix hit rate falls from 80–89% on same-model calls to 44–65%. ShadowPrefill prefills that context on the model the router did not choose, using capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. Across 24 model pairs and five agentic benchmarks, switch-call latency falls by 8.5–21.4% and routed token cost by 22.5–46.1%, normal-traffic SLOs hold, and energy per request rises by 0.4%.
When a per-step router sends an agent's next call to a different model, the target must re-prefill the session context, and the prefix hit rate falls from 80–89% on same-model calls to 44–65%. ShadowPrefill prefills that context on the model the router did not choose, using capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. Across 24 model pairs and five agentic benchmarks, switch-call latency falls by 8.5–21.4% and routed token cost by 22.5–46.1%, normal-traffic SLOs hold, and energy per request rises by 0.4%.
Haoran Qiu（First author）
ShadowPrefill Model Routing Prefix Caching LLM Serving
arXiv Nov 6, 2026
Nebula lifts position-independent caching to the cluster: median time-to-first-token falls 61.4% on average across four models and four workloads
For retrieval-augmented generation and agent workloads, Nebula promotes position-independent caching (PIC) segments, which let the same text segment reuse already-computed key-value cache under a different prefix, into cluster-level cache objects, enabling remote reuse and parallel prefill. Distributed caching systems had reused key-value cache only under strict prefix matching, while single-node PIC could not draw on other nodes' cache or compute. Across four models and four real RAG and agentic workloads, Nebula cuts median time-to-first-token by an average of 61.4% against the strongest baseline and recovers an average of 76.1% of the tokens missed by prefix caching.
For retrieval-augmented generation and agent workloads, Nebula promotes position-independent caching (PIC) segments, which let the same text segment reuse already-computed key-value cache under a different prefix, into cluster-level cache objects, enabling remote reuse and parallel prefill. Distributed caching systems had reused key-value cache only under strict prefix matching, while single-node PIC could not draw on other nodes' cache or compute. Across four models and four real RAG and agentic workloads, Nebula cuts median time-to-first-token by an average of 61.4% against the strongest baseline and recovers an average of 76.1% of the tokens missed by prefix caching.
Yifei Liu（First author）
Nebula distributed LLM serving framework Position-Independent Caching KV Cache Reuse Prefix Caching
researchsquare.com Nov 11, 2026
Coding agents stop rewriting history: ObsDrop drops stale KV instead, reaching 5.37× throughput at 16 concurrent agents
Coding agents mostly append to their history, so prefix-cache reuse reaches 98–99%, yet text-level context compaction rewrites earlier tokens and forces the whole suffix to be recomputed. ObsDrop moves compaction from the prompt into the KV cache: it keeps the logical token prefix unchanged while releasing the KV of stale tool observations and excluding those positions from attention, implemented in SGLang and Sparse-vLLM. At 16 concurrent agents it reaches 2.45–2.85× throughput with FlashInfer and a 5.37× peak with Triton relative to no compaction, with task quality close to no compaction.
Coding agents mostly append to their history, so prefix-cache reuse reaches 98–99%, yet text-level context compaction rewrites earlier tokens and forces the whole suffix to be recomputed. ObsDrop moves compaction from the prompt into the KV cache: it keeps the logical token prefix unchanged while releasing the KV of stale tool observations and excluding those positions from attention, implemented in SGLang and Sparse-vLLM. At 16 concurrent agents it reaches 2.45–2.85× throughput with FlashInfer and a 5.37× peak with Triton relative to no compaction, with task quality close to no compaction.
Pennsylvania State University
Musa Cim（First author, Corresponding author）
ObsDrop Context Compaction Prefix Caching KV Cache Coding Agents
arXiv Nov 7, 2026
Mirage puts a gate on KV-cache timing leaks: probing a sensitive prefix only delays the gateway stream while the GPU keeps reusing cache
Shared prefix caching makes a cache-hit request reach its first token much faster than a miss, and an attacker can exploit that gap to guess another tenant's prompt token by token. Mirage stops isolating caches by content and instead counts new candidate branches under the same sensitive prefix at the gateway; once the budget is spent, it shifts the whole output stream so a hit looks like a natural miss. On SGLang across Qwen, Llama and DeepSeek models and three timing attacks, attack success falls near the random-guessing floor while median first-token latency on public prefixes stays essentially unchanged.
Shared prefix caching makes a cache-hit request reach its first token much faster than a miss, and an attacker can exploit that gap to guess another tenant's prompt token by token. Mirage stops isolating caches by content and instead counts new candidate branches under the same sensitive prefix at the gateway; once the budget is spent, it shifts the whole output stream so a hit looks like a natural miss. On SGLang across Qwen, Llama and DeepSeek models and three timing attacks, attack success falls near the random-guessing floor while median first-token latency on public prefixes stays essentially unchanged.
Xiaobei Yan（First author）
Mirage KV Cache Timing Side Channel Prefix Caching SGLang

You’re all caught up

Model switches no longer throw away the prefix cache: ShadowPrefill warms the unchosen model in idle capacity, cutting switch-call latency by up to 21.4%

Nebula lifts position-independent caching to the cluster: median time-to-first-token falls 61.4% on average across four models and four workloads

Coding agents stop rewriting history: ObsDrop drops stale KV instead, reaching 5.37× throughput at 16 concurrent agents

Mirage puts a gate on KV-cache timing leaks: probing a sensitive prefix only delays the gateway stream while the GPU keeps reusing cache