Agent serving stops restarting nodes to rebalance: Evolving Infra switches prefill and decode in place, cutting a switch from 754 seconds to 6.2
Synopsis
For coding, math and computer-use agent workloads, Evolving Infra decouples compute allocation between prefill and decode from KV cache management: a node switches phase without a restart, and its KV stays readable from a shared pool. On up to 256 GB300 GPUs it cuts mean end-to-end latency and P95 time to first token by up to 4.1 and 14 relative to NVIDIA Dynamo, and a phase switch takes 6.2 seconds on average against 754 seconds for a Dynamo relaunch.
Interpretation
Under agent workloads, an inference cluster can change the split of prefill and decode nodes without relaunching serving workers, and the KV cache of a switching node stays readable by other nodes during the switch. Earlier prefill-decode disaggregated systems tied each node's KV cache to that node's phase, so changing the split either relaunched the worker or stranded the switched node's cache. Serving GLM-5.2 and Kimi K2.6 on 36 GB300 GPUs, Evolving Infra completes a phase switch in 6.2 seconds on average, while a Dynamo relaunch takes 754 seconds and keeps the node out of service throughout.
Every node runs one engine that serves both prefill and decode, the host memory of all nodes forms a shared KV pool, and a cluster-wide directory records how many active branches reference each cached block. The shared pool lets a prefill node pull KV directly from a peer's host memory over RDMA and finish the fetch and the load into GPU memory while the request waits, rather than loading layer by layer once prefill starts. On Claude Code traces with 126 concurrent agents at a fixed 4P5D split, asynchronous fetching cuts idle GPU time of prefill nodes from 23.6% to 14.4% and reaches 1.12 times the throughput of upstream SGLang.
An online planner estimates the throughput of each split with iteration-time models fitted online, separating cached prefixes from uncached tokens, and switches only when the predicted gain clears a statistical threshold. Existing planners rely on coarse signals such as queue depth, KV utilization or time to first token predicted from the full input length, so short bursts from sub-agents starting together trigger switches without sustained gain. In a three-hour run shifting from math to computer-use agents, Evolving Infra reaches 2.14 times the throughput of Dynamo and 1.28 times that of the best static split, lowering mean end-to-end latency by 2.15 and 1.33 relative to them.
Perspective
The demonstrated setting is coding, math and computer-use agents served with GLM-5.2 and Kimi K2.6 on 36 GB300 GPUs, with additional evaluation on 256 GB300 GPUs and in simulation at 3,072 GPUs; branch counts depend on the agent harness tagging each request with a branch ID and reporting when a branch finishes.
Under open-loop replay, Evolving Infra misses its latency targets at 0.75 agents per minute because a single planner switch at minute 75 queues prefill requests until prefill nodes are added back at minute 85; on the decode-heavy Putnam workload the planner keeps only one or two prefill nodes, trading throughput for a P95 time to first token above Dynamo's.