NVIDIA Wires AI-Factory Telemetry Into One Causal Graph: 81 Alerts Become One Incident, Saving About 54 Cluster Hours

The unit of AI-factory operations shifts from a single GPU or switch metric to anomalies along a job's dependency chain. In synchronous distributed training a job advances at the pace of its slowest rank, so one slightly hot GPU, one NVMe device that slows only under sustained load, or one switch path generating a pause-frame storm makes many healthy resources wait. Rule-based monitoring assumes a known metric crossing a known boundary means a known condition, which works for clear failures but misses gray failures where a component stays available and the issue appears only under load or in comparison with peers; tightening thresholds adds warnings during normal workload changes. In one NVIDIA scenario, a page fault forced a network switch to restart, producing RX pause frames propagating between switches, repeated link downs and GPU temperature and utilization anomalies, surfacing as up to 81 noisy events while no single alert explained the slowdown.

The platform places infrastructure, scheduler and workload elements in one dynamic knowledge graph, so an operator can start from an anomalous interface and find its node, the fabric path it serves, current job placement and nearby components in the same failure domain, or start from a job ID and trace GPU, host, NIC, switch and log evidence. Previously network teams saw interface counters and fabric events, compute teams saw node and GPU health, and ML teams saw step times and failed jobs; each view was accurate but the relationships between them stayed hidden, so triage meant dashboard searches and cross-team handoffs. NVIDIA says Minerva processes billions of metric and log data points per hour, with supervised models identifying known critical error patterns and attaching remediation guidance and unsupervised models learning normal behavior for a component and comparing it with peers; log anomaly detection uses NLP to surface previously unseen patterns.

Detections are grouped by a topology-aware correlation engine, so downstream symptoms become one incident carrying affected topology and workloads instead of competing for attention. Previously interface errors, pause frames, congestion indicators, GPU utilization changes and slower jobs from one physical fault sat on separate dashboards, leaving the on-call engineer to correlate alerts and separate signal from noise by hand. NVIDIA says its testing shows significant reductions in notified events and tickets for supported scenarios; in one case Minerva's full-stack job RCA dashboard traced slowdowns to an NVMe device on one host that could not wake from a low-power state, and draining and returning the node saved an estimated 5,530 hours of job runtime.

The demonstrated scope is NVIDIA internal testing and several operations cases covering telemetry from GPUs, NVLink and Spectrum-X switches, SuperNICs, DPUs, storage, NCCL, schedulers and workloads; Minerva is being developed in alignment with NVIDIA DSX OS, and capabilities beginning with the data collector will become open-source parts of DSX OS.

NVIDIA says testing shows significant reductions in notified events and tickets but gives no magnitude or statistical basis; the roughly 54 hours, 5,530 hours and minutes-instead-of-hours figures are individual case narratives. Operators still need other Minerva capabilities to determine root cause, and Sherlock's recommendations lead back to the telemetry and topology that support them.