NVIDIA open-sources its failure attribution engine: one GPU fault stops reading as many NCCL errors
Synopsis
In distributed AI training, one GPU or fabric fault surfaces as NCCL errors on other ranks, scheduler state changes and thousands of downstream messages, while the reported exception only shows where the workload stopped. NVIDIA has open-sourced Attribution, the tool it uses internally, which correlates workload-scoped logs, metrics, telemetry and scheduler state, uses deterministic rules to separate downstream symptoms from the root cause, and returns a structured diagnosis. In a multi-rank training example it named XID 149 on node-02 as an NVLink signal-integrity root cause, kept the other ranks' NCCL errors as symptoms, and reported the workload as a hardware failure.
Interpretation
Failure diagnosis for distributed AI workloads shifts from reading where the exception stopped to returning a root cause, affected nodes and an evidence trail per workload. Previously the reported exception showed where the workload stopped, not where the failure began; a single error could cascade into NCCL errors on other ranks, scheduler state changes and thousands of downstream messages. NVIDIA says the tool has run in production across its AI infrastructure for years, analyzing millions of workloads each week across hundreds of thousands of GPUs; in one large RDMA-backed cluster it correlated port errors, connection resets and scheduler events to patterns around specific host and switch ports.
Attribution works on one workload attempt at a time: scope the workload, collect observations, extract and correlate events, then return a structured result. The path is driven by a core attribution engine, exposed through the Attribution Service over an HTTP API, with the Attribution Blueprint connecting cluster schedulers and telemetry sources and writing the result. In the example, application logs hold three different NCCL errors and node-02 system logs hold XID 149 whose error words decode to an NVLink signal-integrity issue; the rules retain that symptom, demote the other node symptoms, and output a hardware-failure attribution naming node-02.
Rules are deterministic and inspectable, and rule changes ship with the Service rather than being uploaded dynamically through the HTTP API. Deterministic rules make results reproducible and customizable, and structured output lets downstream systems consume terminal_state and sub_domain instead of parsing product-specific logs. The bundled SrunInvalidPyxisArgument rule recognizes an invalid Pyxis argument in an application log and carries a co-located test case; the Service loads the bundled rule catalog at startup and evaluates those inline cases.
Perspective
The tool targets cluster operations and infrastructure teams diagnosing failures within a given workload scope; the sample evidence shipped with the repository runs locally in Docker, while connecting a live cluster requires wiring scheduler and telemetry connectors per the Blueprint configuration reference.
Attribution quality depends on the quality and completeness of the underlying telemetry; when multiple causes fit the evidence, sub_domain is marked ambiguous and the diagnosis records that distinction instead of turning incomplete evidence into a definitive conclusion.