# Failure Attribution

AskALCF, an agentic system for high-performance computing user support, attributes each failed question-answer turn to one of five components—knowledge, retrieval, prompt, model or tools—instead of returning only a quality score. Across 2,174 production support turns it cut unattributable failures from 41.4% to 14.2%, and on a 150-turn expert reference it raised agreement from 108 to 123 turns. The object of improvement is the rules, prompts and review policy rather than model weights, and experts retain authority over every production-facing change.

AskALCF, an agentic system for high-performance computing user support, attributes each failed question-answer turn to one of five components—knowledge, retrieval, prompt, model or tools—instead of returning only a quality score. Across 2,174 production support turns it cut unattributable failures from 41.4% to 14.2%, and on a 150-turn expert reference it raised agreement from 108 to 123 turns. The object of improvement is the rules, prompts and review policy rather than model weights, and experts retain authority over every production-facing change.

In distributed AI training, one GPU or fabric fault surfaces as NCCL errors on other ranks, scheduler state changes and thousands of downstream messages, while the reported exception only shows where the workload stopped. NVIDIA has open-sourced Attribution, the tool it uses internally, which correlates workload-scoped logs, metrics, telemetry and scheduler state, uses deterministic rules to separate downstream symptoms from the root cause, and returns a structured diagnosis. In a multi-rank training example it named XID 149 on node-02 as an NVLink signal-integrity root cause, kept the other ranks' NCCL errors as symptoms, and reported the workload as a hardware failure.

In distributed AI training, one GPU or fabric fault surfaces as NCCL errors on other ranks, scheduler state changes and thousands of downstream messages, while the reported exception only shows where the workload stopped. NVIDIA has open-sourced Attribution, the tool it uses internally, which correlates workload-scoped logs, metrics, telemetry and scheduler state, uses deterministic rules to separate downstream symptoms from the root cause, and returns a structured diagnosis. In a multi-rank training example it named XID 149 on node-02 as an NVLink signal-integrity root cause, kept the other ranks' NCCL errors as symptoms, and reported the workload as a hardware failure.

The work proposes EMFA, which reconstructs failed multi-agent trajectories into structured dependency representations, explicitly models cascading error propagation and persistent interaction loops, and combines dual-branch candidate screening with counterfactual verification to identify the decisive error—the agent–step pair whose correction would recover the failed execution; on the Who&When benchmark it achieves state-of-the-art step-level attribution accuracy, improving the previous best by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, with the best agent-level accuracy on Hand-Crafted and competitive results on Algorithm-Generated.

The work proposes EMFA, which reconstructs failed multi-agent trajectories into structured dependency representations, explicitly models cascading error propagation and persistent interaction loops, and combines dual-branch candidate screening with counterfactual verification to identify the decisive error—the agent–step pair whose correction would recover the failed execution; on the Who&When benchmark it achieves state-of-the-art step-level attribution accuracy, improving the previous best by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, with the best agent-level accuracy on Hand-Crafted and competitive results on Algorithm-Generated.

After quantifying that text embeddings and hidden states separate root-cause steps only weakly, the work proposes ReCast: an attribution-aware probe selects layers, a sparse Johnson–Lindenstrauss projection builds complementary pattern and deviation features, and a bidirectional encoder trained with contrastive and ranking objectives converts frozen-LLM hidden states into attribution-oriented step representations; it also releases the ReCast-2K training set and attains the best Hit@1 on Who&When, TraceElephant, AFTraj-2K, and Who&When Pro.

After quantifying that text embeddings and hidden states separate root-cause steps only weakly, the work proposes ReCast: an attribution-aware probe selects layers, a sparse Johnson–Lindenstrauss projection builds complementary pattern and deviation features, and a bidirectional encoder trained with contrastive and ranking objectives converts frozen-LLM hidden states into attribution-oriented step representations; it also releases the ReCast-2K training set and attains the best Hit@1 on Who&When, TraceElephant, AFTraj-2K, and Who&When Pro.