Model switches no longer throw away the prefix cache: ShadowPrefill warms the unchosen model in idle capacity, cutting switch-call latency by up to 21.4%

When a per-step-routed agent changes models, the new model must re-prefill the session context before it emits a first token, and that wait lands directly on the switch call. On same-model calls, prefix caching reuses 80–89% of prompt tokens; after a switch the target holds no such cache, the hit rate falls to 44–65%, and a switch call takes 1.5–3.2 times as long as a same-model call. The gap is measured on representative agentic workloads, including a production GitHub Copilot coding-agent trace with a mean context of 63K tokens.

ShadowPrefill has the model the router did not choose compute the session context into its own KV cache ahead of time, so a switch only prefills the new tail. Shadow prefill runs as lowest-priority work inside the latency slack that each iteration of a serving instance leaves unused, rather than on extra GPUs. A vLLM prototype is evaluated with LLMRouter as the per-step router on 24 model pairs over five agentic benchmarks, including SWE-bench and the GitHub Copilot coding trace, with normal traffic replayed from Azure inference traces.

Most of the gain comes from shorter time to first token, at the cost of a small energy increase. Switch-call TTFT falls by 44.1% on the benchmarks and 47.0% on coding sessions, and mean latency over all LLM calls falls by 5.0% and 9.7%. In the same evaluation, energy per request rises by 0.4% and normal-traffic SLO attainment does not drop; a dedicated-instance scheme using 25% more GPUs raises energy by 11.7%.

Evaluation covers 24 model pairs and five agentic benchmarks in simulation, plus a vLLM prototype on 16 H100s; normal traffic is replayed from Azure inference traces, with TTFT and TPOT as the SLOs. The prototype validates only the request-level core mechanism and omits the per-iteration credit, the SLO-slack controller, the evict-first shadow KV class, and the host-memory tier.

Shadow prefill is speculative: in the complete design 63% of shadow tokens are never used, and a wrong prediction costs the harvested capacity and its energy. The gain varies with conditions: switch-call latency falls only about 3% for targets above 50B, the all-call latency gain approaches zero at about 0.5 switches per session, and energy rises by 11.5% under prefill/decode disaggregation.