NVIDIA Puts Its Self-Healing Recovery Engine on Kubernetes: Faulty Nodes Get Isolated While Training Jobs Restart Without Waiting for Root Cause

Fault recovery for large AI training clusters moves from covering only Slurm high-performance computing workloads to Kubernetes workloads managed by NVIDIA Run:ai. Previously the autonomous recovery engine in Mission Control was limited to high-performance computing workloads leveraging Slurm, while manual intervention, static checkpointing and reactive monitoring do not scale when dozens of concurrent multi-node workloads share infrastructure. Mission Control 2.5 is available today and existing users can move to the latest version; the release also includes a beta-only AI-enabled network anomaly detection service, improved authentication via central Keycloak, additional security and Kubernetes hardening updates, and early access for NVIDIA Vera Rubin NVL72.

Recovery splits into parallel job and hardware tracks, so a training job can restart before root-cause diagnosis finishes. The autonomous recovery engine consists of autonomous job recovery, which tracks where each job runs and uses failure history and job topology to classify the recovery path, and autonomous hardware recovery, which monitors nodes and GPUs, removes anomalies from scheduling and runs diagnostics or remediation. In a scenario where one GPU starts to fail, with symptoms such as ECC errors, a hung worker or a communication timeout, the job side restarts or reschedules the workload while the hardware side isolates the node, records the reason and starts diagnostics that can continue after the workload is moving again.

The system distinguishes transient failures from hardware failures to avoid quarantining healthy nodes. When a training worker exits, hits a recoverable timeout or briefly loses coordination while the node is healthy, job recovery uses job context to treat the failure as transient and retries according to scheduler policy instead of quarantining hardware that needs no repair. Hardware recovery keeps monitoring in the background but stays out of the recovery path unless new evidence points to hardware, restoring the workload without unnecessary operator noise or shrinking the scheduling pool.

Hardware recovery can also remove degraded nodes from scheduler eligibility before degradation becomes a job failure. A node may show repeated GPU health warnings or connectivity problems that make it risky for future scheduling, and hardware recovery can proactively remove it and start diagnostics before a training job lands there. This mirrors how Mission Control has enabled Slurm to keep faulty nodes out of allocations until they are validated again, giving the Kubernetes-based scheduler a cleaner pool of usable resources.

The recovery engine targets platform teams running multi-node AI training jobs on shared infrastructure, covering Kubernetes workloads managed by NVIDIA Run:ai alongside existing Slurm workloads; the source gives no quantified downtime or repair-time figures.

The source describes recovery behavior through scenarios and reports no measured mean-time-to-repair, mean-time-between-failure or recovery success rates; the network anomaly detection service is beta-only in version 2.5, and Vera Rubin NVL72 support is early access.