Where a KV-cache repair budget should go: recomputing the answer-bearing chunk beats token selection at 5%

Encoding retrieved chunks separately and concatenating their caches makes the model lose context-dependent questions it would otherwise answer, and the loss grows when the other chunks are of the same kind as the answer-bearing chunk. Among questions that full prefill answers and that require the retrieved context, position-corrected reuse loses 19% to 100%; even with off-domain prose as distractors, 11% to 92% of the loss remains. Across twelve settings formed by Qwen3-8B, Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2 on LoCoMo, NQ-Open, TriviaQA and Gorilla, every reuse-loss interval excludes zero, and the extra loss from same-category distractors is significantly positive in nine of the twelve settings.

The recoverable damage is spread across the distractor chunks, so a small budget spent on a few of them recovers little, and the budget is better spent on the chunk that holds the answer. Replacing every distractor's cache with its full-prefill cache recovers more than replacing only the target in all twelve settings, significantly in eleven, whereas replacing a single distractor changes retention significantly in only two of 24 cases. In the nine open-domain QA and API-documentation settings, recomputing the known target chunk at a 5% budget beats the token-level selector in all nine, by 0.10 to 0.49; at a 10% budget it remains significantly ahead in four and is never significantly behind.

Recomputing the target works because it is re-encoded against the reused cache the question actually reads, not because its KV comes closer to the full-prefill KV. Inserting the target's full-prefill KV into the reused cache is beaten by recomputing the target in all seven settings tested, by 0.13 to 0.38, even though the recomputed KV stays at a median relative distance of 0.23 to 0.30 from the full-prefill KV. On 270 questions in seven settings, inserting the target's full-prefill KV raises retention by +0.11 while the earlier chunks stay reused and by +0.25 when their full-prefill KV is restored as well, an interaction of +0.14.

In deployment the target is unknown and question attention over reused caches rarely points to it, so a retriever must locate it, and the retrieved chunk is moved next to the question before recomputation. Over reused caches, question attention selects the target in at most 18.8% of development questions, while it most often selects the last chunk before the question, in 58% to 100% of them. At the same 5% budget, moving the retriever's top chunk next to the question and recomputing it there significantly beats InfoFlow-style reordering in six of the nine open-domain QA and API-documentation settings; on dialogue memory, chunk-level recomputation never significantly improves on the token-level selector chosen for each setting.

The experiments use 7B-8B instruction-tuned models on four English datasets, with the main conditions repeated on Qwen3-14B and Qwen3-32B; chunk-level allocation needs either the known target or a retriever, so deployment gains depend on retrieval quality. Only small budgets counted in recomputed tokens are studied, and latency is measured only in the authors' research engine.

The gains are bounded by retrieval: when the retriever does not rank the target first, recomputing the moved chunk does worse than recomputing selected tokens, and the gain from moving the top three chunks falls within the prediction from the retriever's hit rate in seven of nine settings. At a 10% budget token-level selectors remain competitive, and on dialogue memory, for the 7B-8B models, chunk-level recomputation never significantly improves on the token-level selector chosen for each setting.