# Model Routing

When a per-step router sends an agent's next call to a different model, the target must re-prefill the session context, and the prefix hit rate falls from 80–89% on same-model calls to 44–65%. ShadowPrefill prefills that context on the model the router did not choose, using capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. Across 24 model pairs and five agentic benchmarks, switch-call latency falls by 8.5–21.4% and routed token cost by 22.5–46.1%, normal-traffic SLOs hold, and energy per request rises by 0.4%.

When a per-step router sends an agent's next call to a different model, the target must re-prefill the session context, and the prefix hit rate falls from 80–89% on same-model calls to 44–65%. ShadowPrefill prefills that context on the model the router did not choose, using capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. Across 24 model pairs and five agentic benchmarks, switch-call latency falls by 8.5–21.4% and routed token cost by 22.5–46.1%, normal-traffic SLOs hold, and energy per request rises by 0.4%.

General-purpose AI systems are shifting from a single model to orchestrating many specialized models, tools and policies, but routing had mostly been studied on language models. Route Anything builds a unified trajectory matrix of 25 benchmarks, 452 candidate configurations and 374,401 scored runs, together with a routing theory for quality–cost trade-offs. On 19,814 test inputs the per-input oracle beats the best fixed candidate by 5.38 points; learned routers beat it by 0.3 points across 16 benchmarks, by 2.1 points on video generation, and by 1.1 points on robot manipulation with 24% less policy time.

General-purpose AI systems are shifting from a single model to orchestrating many specialized models, tools and policies, but routing had mostly been studied on language models. Route Anything builds a unified trajectory matrix of 25 benchmarks, 452 candidate configurations and 374,401 scored runs, together with a routing theory for quality–cost trade-offs. On 19,814 test inputs the per-input oracle beats the best fixed candidate by 5.38 points; learned routers beat it by 0.3 points across 16 benchmarks, by 2.1 points on video generation, and by 1.1 points on robot manipulation with 24% less policy time.

Web agents edit images by calling open editing models in a critique-and-retry loop, yet a retry after a rejection succeeds only 17–53% as often as a first try, while existing routers treat each retry as an independent draw. RetryRoute estimates each model's next-attempt success probability from the request's failure history, plans retries, escalations and abandonment over failure states, and charges live queue waits to the cost. At the main budget it exceeds the strongest of eleven prior baselines by 2.8 points on average over eight groups, and on a three-GPU serving testbed one setting raises the share of edits meeting the service objective by 5.0–6.3 points over the best prior baseline at every load.

Web agents edit images by calling open editing models in a critique-and-retry loop, yet a retry after a rejection succeeds only 17–53% as often as a first try, while existing routers treat each retry as an independent draw. RetryRoute estimates each model's next-attempt success probability from the request's failure history, plans retries, escalations and abandonment over failure states, and charges live queue waits to the cost. At the main budget it exceeds the strongest of eleven prior baselines by 2.8 points on average over eight groups, and on a three-GPU serving testbed one setting raises the share of edits meeting the service objective by 5.0–6.3 points over the best prior baseline at every load.