A million-token context on 16 H200s: GLM-5.3 shards the KV cache by token and lifts per-user speed up to 3.2×

Under long-context agentic workloads, GLM-5.3's weights take up most of each H200's memory, leaving little room for the KV cache, and under load the prefill engines can become the bottleneck for the whole deployment. Previously, Hybrid HiSparse let long contexts fit in an aggregated deployment on a single 8× H200 node; a prefill/decode disaggregated deployment needs more than that. A 1M-token GLM-5.3 request needs 53.4 GiB of FP8 KV, while each GPU has room for only about 15 GiB.

The prefill engine splits each prompt across eight GPUs by token rather than by attention head, so every GPU computes the queries for one eighth of the prompt with all 64 heads. DP8 assigns each prompt to a single GPU, so its attention cannot use the other seven; TP8 partitions heads, but MLA's latent KV is shared by all 64 heads, so every GPU still stores the full cache, and the indexer is not partitioned at all. With the token split, each KV row read serves all 64 heads, raising sparse MLA from about 26 to about 191 FLOPs per byte moved, close to where the H200 becomes compute-bound in BF16.

DCP shards the KV cache itself round-robin by token, so each GPU stores every eighth token and a 1M-token request drops to 6.7 GiB per GPU. PCP spreads query work but all-gathers the new KV every layer, so every GPU still ends up storing the full cache, and that replicated cache is what limits context length. At equal cached context, DCP8 adds about 6% per step up to 64K tokens of context, rising to 15% at 248K; beyond roughly 250K tokens, PCP8 alone cannot hold the KV cache.

The deployment runs on two 8× H200 nodes, with one PCP8 × DCP8 × EP8 prefill engine and eight DP decode ranks with EP8 and Hybrid HiSparse. It is compared against DP8 prefill (eight DP ranks with 64 GiB of CPU KV offload each, the same 512 GiB in total) and offload-only decode, with every configuration using the same 16 GPUs, host memory and vLLM build. On the SemiAnalysis AgentX benchmark, up to c32 the deployment is mostly prefill-bound, and PCP8 × DCP8 prefill gives 1.7–3.2× the interactivity of DP8 at the same throughput while cutting median TTFT from 1.5–5.2 s to 0.8–0.9 s; at c64 HiSparse decode nearly doubles throughput, to 8,340 logical tok/s/GPU at 29.6 tok/s/user.

The results come from a prefill/decode disaggregated deployment on two 8× H200 nodes, using an FP8 KV cache and MTP3 speculative decoding, measured on the SemiAnalysis AgentX benchmark by replaying Claude Code traces with a fixed synthetic acceptance length of 2.99; every configuration uses the same 16 GPUs, host memory and vLLM build.

DCP all-gathers the whole cached context every layer, so its cost grows with context length, about 6% up to 64K tokens and 15% at 248K; that overhead could be reduced by overlapping each layer's all-gather with the previous layer's compute, an optimization still in progress.