NVIDIA open-sources cluster-level training recovery: nearly 200 jobs a day, 69-second detection and 94-second action on average
Synopsis
As distributed training scales, single-GPU failures compound across a cluster, and per-job manual log analysis, fault isolation and checkpoint restarts no longer cover every job. NVIDIA has open-sourced NVIDIA Recovery under Apache 2.0, a workload-level service that combines application logs, GPU metrics and scheduler signals through four stages, detect, attribute, isolate and restart or stop, and talks to Slurm through an Advisor API. On one training cluster, monitoring from July 9 to October 9 covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.
Interpretation
Cluster-level training recovery moves from an engineer handling each job by hand to a continuously running control loop that coordinates across workloads. Previously each interruption demanded log analysis, fault isolation and a restart from a saved checkpoint, while affected GPUs sat idle during recovery; as job count and size grow, overlapping failures add engineering burden and waste GPU-hours. On one training cluster, monitoring from July 9 to October 9 covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.
The recovery loop runs in four stages, detect, attribute, isolate, and restart or stop, with each action set by the interruption and the configured policy. The anomaly detector combines application logs with scheduler signals to identify crashes, hangs and stragglers, recognizing PyTorch and NCCL failures, CUDA errors and interconnect failure patterns; missing progress signals or known communication-hang patterns indicate hangs, and sustained throughput degradation against a recent baseline indicates stragglers. The service runs as a control-plane service and communicates with the cluster's scheduler integration through an HTTP or gRPC API; it supports Slurm, with Kubernetes support on the roadmap, and recovery recommendations follow a common API contract that a cluster-side controller translates into workload-manager actions, so application code does not change.
Isolation excludes implicated nodes from later restart attempts and stops the workload when further retries would be unproductive, with a watchdog as a safety net. Attribution classifies the interruption from the scheduler signals and application logs gathered during detection, and optional integration with NVIDIA Attribution adds deeper diagnosis to distinguish hardware faults, system-software issues and application bugs; that evidence informs node exclusion and enriches the audit trail. The anomaly remediator tracks implicated nodes across attempts to avoid recurring failures, and if a recovery decision takes too long or fails outright, a watchdog service lets normal scheduling proceed so workloads do not wait indefinitely.
Perspective
The demonstrated scope is recovery of Slurm jobs, including Slurm on Kubernetes through slurm-operator; the service deploys as a control-plane service, integrates with the scheduler over HTTP or gRPC, and new deployments start in monitor mode, publishing advice without taking scheduler actions. IBM Research ran monitor mode first on its Slurm cluster, compared the advice with its own manual recovery decisions, and then enabled autonomous action; Verda deployed on VictoriaLogs rather than Loki and contributed a VictoriaLogs connector.
Recovery-action latency ends at the automated action itself; scheduling and checkpoint restoration determine when productive training actually resumes. This version recovers Slurm jobs, while recovery for Slurm job arrays and native Kubernetes recovery starting with JobSet and Kubeflow PyTorchJob are listed as next capabilities; sharper failure classification, a fuller record of each recovery decision in the dashboard and API, and work toward a stable release are also on the roadmap.