NVIDIA Puts Its Self-Healing Recovery Engine on Kubernetes: Faulty Nodes Get Isolated While Training Jobs Restart Without Waiting for Root Cause
NVIDIA Mission Control 2.5 extends an autonomous recovery engine that previously served only Slurm high-performance computing workloads to Kubernetes workloads managed by NVIDIA Run:ai. Such AI factories had relied on manual intervention, static checkpointing and reactive monitoring, which does not scale to dozens of concurrent multi-node training jobs. The release splits recovery into three coordinated layers: workload placement, job recovery that uses failure history and job topology to choose restart, retry or requeue, and hardware recovery that isolates suspect nodes and runs diagnostics, so recovery need not wait for root-cause analysis.