Hyesung Jeon

The work presents PReCache, a training-free KV cache sharing framework with two designs: PreLRShared precomputes a compact agent-specific low-rank cache for every agent when each segment of the shared context is first processed, removing later agents' repeated prefill over the accumulated trajectory, while ReBaseShared further reconstructs the shared base cache from adapter-free hidden states to reduce dependence on the previous agent, with lazy prefill for single-stream inference and double batching for concurrent serving; across LLaMA-3.

The work presents PReCache, a training-free KV cache sharing framework with two designs: PreLRShared precomputes a compact agent-specific low-rank cache for every agent when each segment of the shared context is first processed, removing later agents' repeated prefill over the accumulated trajectory, while ReBaseShared further reconstructs the shared base cache from adapter-free hidden states to reduce dependence on the previous agent, with lazy prefill for single-stream inference and double batching for concurrent serving; across LLaMA-3.

The work presents KVCMAS, an online KV cache correction framework for prompt-specialized multi-agent systems that represents cross-agent cache deviations as compact low-rank states and chains corrections along the agent workflow without an additional context-free reference prefill, matching or improving the accuracy of prior KV cache sharing methods across language and vision-language workloads while achieving a 2.0 TTFT speedup over inference without KV cache sharing at 32K shared tokens and 8 QPS and reducing peak GPU memory by up to 3.7 relative to the prior correction method KVComm.

The work presents KVCMAS, an online KV cache correction framework for prompt-specialized multi-agent systems that represents cross-agent cache deviations as compact low-rank states and chains corrections along the agent workflow without an additional context-free reference prefill, matching or improving the accuracy of prior KV cache sharing methods across language and vision-language workloads while achieving a 2.0 TTFT speedup over inference without KV cache sharing at 32K shared tokens and 8 QPS and reducing peak GPU memory by up to 3.7 relative to the prior correction method KVComm.