# AgentX

3 related articles

Subscribe to updates

KVCache.AI Nov 8, 2026
AMD's ATOM inference engine gains a shared Mooncake KV pool, roughly doubling throughput over split pools on four MI350X servers
Coding agents resend the same long conversation, so whether the server still holds that history outside GPU memory decides how many tokens each turn must recompute. AMD's open-source ATOM inference engine, which places prefill and decode on separate GPUs, now connects to Mooncake Store to merge both halves of host memory into one KV pool that prefill saves to and restores from. On four AMD Instinct MI350X servers, the shared pool reached about twice the throughput of per-server pools at 96 and 112 concurrent sessions, with KV reuse held at 94–96%.
Coding agents resend the same long conversation, so whether the server still holds that history outside GPU memory decides how many tokens each turn must recompute. AMD's open-source ATOM inference engine, which places prefill and decode on separate GPUs, now connects to Mooncake Store to merge both halves of host memory into one KV pool that prefill saves to and restores from. On four AMD Instinct MI350X servers, the shared pool reached about twice the throughput of per-server pools at 96 and 112 concurrent sessions, with KV reuse held at 94–96%.
Mooncake Store KV Cache Reuse AgentX
LMSYS Org Sep 16, 2026
SGLang stores KV cache in 4 bits: long-context decode throughput rises 26–30%, and FP8 stalls first under agent concurrency
SGLang adds NVFP4 storage to its paged KV cache, cutting per-token key-value bytes to about 56% of FP8. On Qwen3.8-27B at 32K, 160K and 1M input lengths with matched decode concurrency, peak-batch decode throughput rises about 26–30%; on agentic load beyond concurrency 12, FP8 throughput collapses while NVFP4 keeps climbing. Accuracy gaps vary by task, reaching 1.60 points on SWE-bench Verified.
SGLang adds NVFP4 storage to its paged KV cache, cutting per-token key-value bytes to about 56% of FP8. On Qwen3.8-27B at 32K, 160K and 1M input lengths with matched decode concurrency, peak-batch decode throughput rises about 26–30%; on agentic load beyond concurrency 12, FP8 throughput collapses while NVFP4 keeps climbing. Accuracy gaps vary by task, reaching 1.60 points on SWE-bench Verified.
NVIDIA Qwen
KV Cache Quantization NVFP4 SGLang KV Cache AgentX
vLLM Blog Oct 16, 2026
Eight AMD MI355X GPUs match NVIDIA B300 on Kimi K3: 24.4% more tokens per GPU, 87.4% more per dollar
On the public AgentX agent workload, an eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM reaches about 7,567 total tokens/s/GPU at 50 tokens/s/user P90 interactivity, 24.4% above NVIDIA HGX B300's 6,083; under SemiAnalysis's modeled $1.50 versus $2.26 per GPU-hour, that is 18.16 million versus 9.69 million tokens per dollar of TCO, 87.4% more. The result comes from sustained AMD and vLLM community engineering across attention, recurrent state, MoE and caching rather than a single change.
On the public AgentX agent workload, an eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM reaches about 7,567 total tokens/s/GPU at 50 tokens/s/user P90 interactivity, 24.4% above NVIDIA HGX B300's 6,083; under SemiAnalysis's modeled $1.50 versus $2.26 per GPU-hour, that is 18.16 million versus 9.69 million tokens per dollar of TCO, 87.4% more. The result comes from sustained AMD and vLLM community engineering across attention, recurrent state, MoE and caching rather than a single change.
Kimi K3 AgentX Speculative Decoding Decode Context Parallelism Tensor parallelism

You’re all caught up

AMD's ATOM inference engine gains a shared Mooncake KV pool, roughly doubling throughput over split pools on four MI350X servers

SGLang stores KV cache in 4 bits: long-context decode throughput rises 26–30%, and FP8 stalls first under agent concurrency

Eight AMD MI355X GPUs match NVIDIA B300 on Kimi K3: 24.4% more tokens per GPU, 87.4% more per dollar