# Kubernetes

At KubeCon + CloudNativeCon, NVIDIA said it is donating OpenShell, an open source runtime for securing and governing autonomous agents, to the Cloud Native Computing Foundation (CNCF) as a Sandbox project, while also contributing to Google-led Agent Substrate and releasing three new components of the DSX AI Factory Platform. OpenShell applies deterministic controls to agent activity so teams can enforce policies as agents use tools, access inference services and execute workloads, and its extensible architecture connects their own compute, inference providers, credential storage and monitoring tools. NVIDIA says hundreds of organizations have put OpenShell to use and the project has received tens of thousands of contributions in less than a year since launch.

At KubeCon + CloudNativeCon, NVIDIA said it is donating OpenShell, an open source runtime for securing and governing autonomous agents, to the Cloud Native Computing Foundation (CNCF) as a Sandbox project, while also contributing to Google-led Agent Substrate and releasing three new components of the DSX AI Factory Platform. OpenShell applies deterministic controls to agent activity so teams can enforce policies as agents use tools, access inference services and execute workloads, and its extensible architecture connects their own compute, inference providers, credential storage and monitoring tools. NVIDIA says hundreds of organizations have put OpenShell to use and the project has received tens of thousands of contributions in less than a year since launch.

NVIDIA has donated Agent Substrate, an open-source runtime for running large numbers of sandboxed AI agents on Kubernetes, to the CNCF Sandbox. Agents spend most of their time waiting on a prompt, a tool result or a model response; keeping one Pod per session holds memory for idle sessions, while creating Pods on demand pays a multi-second cold start at every step and loses state. Agent Substrate multiplexes many sessions over a small pool of warm worker Pods, checkpointing idle sessions including memory and filesystem and restoring them onto whichever worker is free, and the project reports 10x higher density than standard containers.

NVIDIA has donated Agent Substrate, an open-source runtime for running large numbers of sandboxed AI agents on Kubernetes, to the CNCF Sandbox. Agents spend most of their time waiting on a prompt, a tool result or a model response; keeping one Pod per session holds memory for idle sessions, while creating Pods on demand pays a multi-second cold start at every step and loses state. Agent Substrate multiplexes many sessions over a small pool of warm worker Pods, checkpointing idle sessions including memory and filesystem and restoring them onto whichever worker is free, and the project reports 10x higher density than standard containers.

NVIDIA has open-sourced NVIDIA Snapshot, a checkpoint-and-restore component for GPU inference workloads on Kubernetes that captures a fully initialized workload, including CPU and GPU memory, as a portable artifact and restores it on nodes with the same GPU model and driver, skipping weight loading, compilation and graph capture. Across vLLM, SGLang and TensorRT LLM, from a 600-million-parameter model to mixture-of-experts models with 1 TB checkpoints on eight GPUs, every tested configuration restored a replica in under 45 seconds, 8x to 36x faster than a cold start.

NVIDIA has open-sourced NVIDIA Snapshot, a checkpoint-and-restore component for GPU inference workloads on Kubernetes that captures a fully initialized workload, including CPU and GPU memory, as a portable artifact and restores it on nodes with the same GPU model and driver, skipping weight loading, compilation and graph capture. Across vLLM, SGLang and TensorRT LLM, from a 600-million-parameter model to mixture-of-experts models with 1 TB checkpoints on eight GPUs, every tested configuration restored a replica in under 45 seconds, 8x to 36x faster than a cold start.

NVIDIA built the AI Factory Operations Agent for its own DGX SuperPOD operations so site reliability engineers no longer have to check compute, network, hardware and scheduling dashboards one by one. The agent is a set of domain-specific sub-agents that communicate over MCP, detect failures in production, correlate root causes across layers and surface recommendations through a chat interface. The framework runs as a Kubernetes control-plane application, its tool calls are constrained by read-only credentials and RBAC, and the orchestration framework is released as experimental open source software.

NVIDIA built the AI Factory Operations Agent for its own DGX SuperPOD operations so site reliability engineers no longer have to check compute, network, hardware and scheduling dashboards one by one. The agent is a set of domain-specific sub-agents that communicate over MCP, detect failures in production, correlate root causes across layers and surface recommendations through a chat interface. The framework runs as a Kubernetes control-plane application, its tool calls are constrained by read-only credentials and RBAC, and the orchestration framework is released as experimental open source software.

NVIDIA CUDA 13.4 adds an application-transparent time-sharing mode that confines each workload to a fraction of GPU memory, so an allocation above the requested size fails as it would on a smaller GPU. KAI Scheduler offers an NvFractions sharing mode from v0.18.0 that picks GPU slices and writes memory and compute mode into Pod annotations, and the new open-source gpu-fractioning operator enforces those limits on each node and measures per-fraction use.

NVIDIA CUDA 13.4 adds an application-transparent time-sharing mode that confines each workload to a fraction of GPU memory, so an allocation above the requested size fails as it would on a smaller GPU. KAI Scheduler offers an NvFractions sharing mode from v0.18.0 that picks GPU slices and writes memory and compute mode into Pod annotations, and the new open-source gpu-fractioning operator enforces those limits on each node and measures per-fraction use.

NVIDIA Mission Control 2.5 extends an autonomous recovery engine that previously served only Slurm high-performance computing workloads to Kubernetes workloads managed by NVIDIA Run:ai. Such AI factories had relied on manual intervention, static checkpointing and reactive monitoring, which does not scale to dozens of concurrent multi-node training jobs. The release splits recovery into three coordinated layers: workload placement, job recovery that uses failure history and job topology to choose restart, retry or requeue, and hardware recovery that isolates suspect nodes and runs diagnostics, so recovery need not wait for root-cause analysis.

NVIDIA Mission Control 2.5 extends an autonomous recovery engine that previously served only Slurm high-performance computing workloads to Kubernetes workloads managed by NVIDIA Run:ai. Such AI factories had relied on manual intervention, static checkpointing and reactive monitoring, which does not scale to dozens of concurrent multi-node training jobs. The release splits recovery into three coordinated layers: workload placement, job recovery that uses failure history and job topology to choose restart, retry or requeue, and hardware recovery that isolates suspect nodes and runs diagnostics, so recovery need not wait for root-cause analysis.

NVIDIA released NVCRE (Cluster Readiness Engine), an open-source Kubernetes controller that runs real distributed workloads (NCCL communication, DCGM level-4 diagnostics, NeMo Nemotron 5 pretraining), measures results across topology-aware node groups, and reports exactly which nodes failed and why, verifying GPU cluster readiness before production workloads land instead of hand-writing NCCL manifests and bisecting racks manually.

NVIDIA released NVCRE (Cluster Readiness Engine), an open-source Kubernetes controller that runs real distributed workloads (NCCL communication, DCGM level-4 diagnostics, NeMo Nemotron 5 pretraining), measures results across topology-aware node groups, and reports exactly which nodes failed and why, verifying GPU cluster readiness before production workloads land instead of hand-writing NCCL manifests and bisecting racks manually.