# NVIDIA Recovery

As distributed training scales, single-GPU failures compound across a cluster, and per-job manual log analysis, fault isolation and checkpoint restarts no longer cover every job. NVIDIA has open-sourced NVIDIA Recovery under Apache 2.0, a workload-level service that combines application logs, GPU metrics and scheduler signals through four stages, detect, attribute, isolate and restart or stop, and talks to Slurm through an Advisor API. On one training cluster, monitoring from July 9 to October 9 covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.

As distributed training scales, single-GPU failures compound across a cluster, and per-job manual log analysis, fault isolation and checkpoint restarts no longer cover every job. NVIDIA has open-sourced NVIDIA Recovery under Apache 2.0, a workload-level service that combines application logs, GPU metrics and scheduler signals through four stages, detect, attribute, isolate and restart or stop, and talks to Slurm through an Advisor API. On one training cluster, monitoring from July 9 to October 9 covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.