AskALCF assigns each HPC support failure to a component: unattributed turns fall from 41.4% to 14.2% across 2,174
Synopsis
AskALCF, an agentic system for high-performance computing user support, attributes each failed question-answer turn to one of five components—knowledge, retrieval, prompt, model or tools—instead of returning only a quality score. Across 2,174 production support turns it cut unattributable failures from 41.4% to 14.2%, and on a 150-turn expert reference it raised agreement from 108 to 123 turns. The object of improvement is the rules, prompts and review policy rather than model weights, and experts retain authority over every production-facing change.
Interpretation
AskALCF turns a failed HPC support turn into a work item with an owner: missing documentation, retrieval that did not return the answer, an inadequate prompt, a wrong model answer, or a failed tool call, each mapped to a team that can act. Earlier HPC question-answering systems mainly produced answers, and quality evaluation gave a single score that could not say who should change what. On 1,229 sessions and 2,174 scored turns, the rule tree left 900 turns (41.4%) unattributed before improvement and 309 (14.2%) afterward, raising attributed turns from 924 to 1,295.
Attribution rests on two retrieval judgments kept apart: whether the passages returned in production were sufficient, and whether the needed information exists anywhere in the documentation corpus. Asking only whether retrieval was sufficient conflates a documentation gap with a retrieval failure, and those two are repaired by documentation authors and retrieval engineers respectively. An ordered tree of 28 rules checks tool and service failures first, then facility scope and clarification, and only then the sufficiency pivot; on the 150-turn expert reference agreement rose from 108/150 to 123/150, and on the session-disjoint 52-turn evaluation split from 35/52 to 40/52.
The diagnostic pipeline is improved by editing the rule tree, judge skills and override gate—the surrounding software—rather than model parameters. Earlier self-improving systems mostly updated internal memory or optimized prompts directly, without a frozen reference and controls to separate real gains from run-to-run variation. Sixteen harness arms were evaluated against the frozen expert reference, with three accepted and nine rejected; a semantically neutral edit still changed 9 and 11 of 171 scored routes in two replicates, so a change is accepted only when pre-registered targets improve and controls stay stable.
Perspective
The demonstrated setting is the production question-answering and ticket workflow for the Polaris and Aurora systems at the Argonne Leadership Computing Facility, with diagnoses reviewed by experts and repair effects measured in offline replay experiments. No generated knowledge change has been merged into the deployed corpus and re-indexed, and three of the five failure routes have not been exercised through an end-to-end repair path.
All labelled data come from ALCF, about two thirds of the 150-turn reference received only one expert label, and the unbalanced class distribution makes per-route estimates unreliable for rare model, unknown and tools_mcp cases. The gate-policy comparison is the only agentic-versus-deterministic control, each repair study represents one experimental campaign, and null arms estimate run-to-run variation but do not substitute for repeated independent trials.