# NVIDIA Dynamo

NVIDIA has donated Agent Substrate, an open-source runtime for running large numbers of sandboxed AI agents on Kubernetes, to the CNCF Sandbox. Agents spend most of their time waiting on a prompt, a tool result or a model response; keeping one Pod per session holds memory for idle sessions, while creating Pods on demand pays a multi-second cold start at every step and loses state. Agent Substrate multiplexes many sessions over a small pool of warm worker Pods, checkpointing idle sessions including memory and filesystem and restoring them onto whichever worker is free, and the project reports 10x higher density than standard containers.

NVIDIA has donated Agent Substrate, an open-source runtime for running large numbers of sandboxed AI agents on Kubernetes, to the CNCF Sandbox. Agents spend most of their time waiting on a prompt, a tool result or a model response; keeping one Pod per session holds memory for idle sessions, while creating Pods on demand pays a multi-second cold start at every step and loses state. Agent Substrate multiplexes many sessions over a small pool of warm worker Pods, checkpointing idle sessions including memory and filesystem and restoring them onto whichever worker is free, and the project reports 10x higher density than standard containers.

NVIDIA has open-sourced NVIDIA Snapshot, a checkpoint-and-restore component for GPU inference workloads on Kubernetes that captures a fully initialized workload, including CPU and GPU memory, as a portable artifact and restores it on nodes with the same GPU model and driver, skipping weight loading, compilation and graph capture. Across vLLM, SGLang and TensorRT LLM, from a 600-million-parameter model to mixture-of-experts models with 1 TB checkpoints on eight GPUs, every tested configuration restored a replica in under 45 seconds, 8x to 36x faster than a cold start.

NVIDIA has open-sourced NVIDIA Snapshot, a checkpoint-and-restore component for GPU inference workloads on Kubernetes that captures a fully initialized workload, including CPU and GPU memory, as a portable artifact and restores it on nodes with the same GPU model and driver, skipping weight loading, compilation and graph capture. Across vLLM, SGLang and TensorRT LLM, from a 600-million-parameter model to mixture-of-experts models with 1 TB checkpoints on eight GPUs, every tested configuration restored a replica in under 45 seconds, 8x to 36x faster than a cold start.

NVIDIA reframes AI-factory memory from how much capacity a GPU has to how many useful tokens each byte produces, combining NVFP4 weight and KV-cache compression, wide expert parallelism for MoE weights, Dynamo KV-cache reuse, Vera CPU staging of context during tool calls, and CMX pod-level context storage. On SemiAnalysis AgentX, which replays recorded coding-agent sessions, Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user, with both at 288 GB per GPU.

NVIDIA reframes AI-factory memory from how much capacity a GPU has to how many useful tokens each byte produces, combining NVFP4 weight and KV-cache compression, wide expert parallelism for MoE weights, Dynamo KV-cache reuse, Vera CPU staging of context during tool calls, and CMX pod-level context storage. On SemiAnalysis AgentX, which replays recorded coding-agent sessions, Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user, with both at 288 GB per GPU.