NVIDIA hands DGX SuperPOD operations to AI agents: cross-layer root-cause hunting, read-only access first
Synopsis
NVIDIA built the AI Factory Operations Agent for its own DGX SuperPOD operations so site reliability engineers no longer have to check compute, network, hardware and scheduling dashboards one by one. The agent is a set of domain-specific sub-agents that communicate over MCP, detect failures in production, correlate root causes across layers and surface recommendations through a chat interface. The framework runs as a Kubernetes control-plane application, its tool calls are constrained by read-only credentials and RBAC, and the orchestration framework is released as experimental open source software.
Interpretation
NVIDIA moved day-to-day AI factory operations from manual layer-by-layer inspection to a team of AI agents, letting site reliability engineers start investigations, switch between them and track every action from a single chat interface. As GPU clusters scaled, operations spanned compute, network, hardware and scheduling, and engineers often had to examine multiple dashboards to identify the cause of a training performance drop. The agent runs on NVIDIA's internal DGX H100, DGX B200 and DGX GB300 production clusters, where it detects failures, correlates root causes across layers and surfaces recommendations.
A user-facing agent runtime receives engineers' natural-language queries and invokes the appropriate domain agents and tools, with sub-agents communicating through MCP, a protocol that defines how agents call external tools. Domain agents connect to infrastructure through OpenClaw plugins and MCP servers, each MCP exposing a focused set of approved operational tools, so adding new agents and integrations does not require changing existing agents. The internal deployment uses job recovery, hardware, observability, research, Kubernetes and cluster management agents, covering multi-layer root-cause analysis of training job failures and goodput degradation, device-level faults such as PCIe and ECC errors, historical Prometheus and Grafana Loki log queries, documentation search and read-only cluster state inspection.
Security is enforced in layers: Kubernetes inspection uses the server-side run_kubectl tool, which invokes a pinned kubectl binary against registered read-only credentials, and the OpenShell runtime holds neither kubectl nor Kubernetes credentials. Commands do not run in the main gateway pod; OpenShell creates a per-session Kubernetes sandbox whose lifecycle does not modify existing services or production workloads. Permission checks show the agent can get pods and configmaps but cannot get secrets or create or update pods.
Perspective
The demonstrated scope is deployment on NVIDIA's internal DGX H100, DGX B200 and DGX GB300 production clusters and Kubernetes inspection under read-only permissions; the orchestration framework is offered as experimental open source software whose APIs, configuration and deployment requirements may change between releases.
NVIDIA is still refining the framework by measuring root-cause-analysis accuracy and tracking SRE agreement, and the material gives no values for those metrics; the open-source portion is experimental, so interfaces and deployment requirements may change.