Haoran Qiu

When a per-step router sends an agent's next call to a different model, the target must re-prefill the session context, and the prefix hit rate falls from 80–89% on same-model calls to 44–65%. ShadowPrefill prefills that context on the model the router did not choose, using capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. Across 24 model pairs and five agentic benchmarks, switch-call latency falls by 8.5–21.4% and routed token cost by 22.5–46.1%, normal-traffic SLOs hold, and energy per request rises by 0.4%.

When a per-step router sends an agent's next call to a different model, the target must re-prefill the session context, and the prefix hit rate falls from 80–89% on same-model calls to 44–65%. ShadowPrefill prefills that context on the model the router did not choose, using capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. Across 24 model pairs and five agentic benchmarks, switch-call latency falls by 8.5–21.4% and routed token cost by 22.5–46.1%, normal-traffic SLOs hold, and energy per request rises by 0.4%.