Customer-support guidance without a re-ranker: a portable skill hits 99.2% procedural equivalence on 132 real cases

Generating customer-support resolution guidance no longer has to depend on a complex retrieval stack; it can be packaged as a portable skill: a natural-language operating manual, a few deterministic scripts, a configuration file, and a corpus with a navigation layer, executed by a general-purpose agent harness. The prior production approach was a retrieval-augmented generation stack with a fine-tuned or prompt-specialized generator, multi-stage retrieval, learned re-rankers and query rewriters, and per-workflow heuristics; such stacks are expensive to build and hard to maintain as the knowledge base, contact-reason taxonomy, and user behavior drift. On 132 real customer-support cases sampled from deployed traffic, an LLM judge rated the skill's guidance as procedurally equivalent to human-approved ground truth on 99.2% of cases, with a 95% confidence interval of [95.8, 99.9].

The skill organizes the retrieval surface as a navigation layer generated automatically by an LLM; retrieval lands on navigation pages and then dereferences stable identifiers to the underlying source articles for evidence. The navigation layer is not a hand-curated summary index: a single GPT-4.1 pass over the 2,881 source articles builds it through an update-or-create loop, so merged entities and typed links emerge without a separate clustering step. An independent, stronger model judged all 3,168 navigation-to-source links against the articles they cite: about 96% were relevant and source-supported, roughly 4% were dangling, irrelevant, or insufficiently supported and were corrected in a follow-up QA pass, and a structural check confirmed every one of the 2,881 source articles is referenced by at least one navigation page.

The evidence contract is enforced by code rather than by prompting: every emitted snippet is checked at save time to be a verbatim substring of its cited article, and non-conformant writes are rejected. The prior practice typically states such constraints in a prompt and relies on model behavior, which cannot be audited independently of the model. In the reported run, 1,769 snippets were emitted across the 132 cases; 99.7% were byte-identical to their cited article before normalization and 100% satisfied the enforced, normalization-tolerant contract, with the residual 0.3% differing only by a curly-versus-straight apostrophe.

Swapping the retrieval backend from the skill's own hybrid search to the organization's production retrieval-augmented generation service leaves procedural equivalence close but separates the two on evidence sufficiency. Both backends search the same corpus, so the needed text is reachable by either; what differs is the workflow, because the navigation layer states what each workflow solves and links to the exact underlying articles, letting the agent issue a targeted follow-up search once it identifies missing evidence, whereas the scripted production workflow completed retrieval before composition and could not re-retrieve. Holding the corpus, the 132 cases, and the evaluation protocol fixed, the wiki backend reached 99.2% procedural equivalence and 83.3% evidence sufficiency, while the production RAG service reached 96.2% and 59.8%, with non-overlapping 95% Wilson intervals on evidence sufficiency.

The skill targets a mature, well-curated operational corpus, a latency-tolerant workflow, and a setting where every case is reviewed by a human, and the evaluation covers 132 real customer-support cases sampled from deployed traffic. The authors explicitly do not claim production pipelines are obsolete and state that the skill says nothing about latency, cost-per-case, or robustness under load. They propose that any task with a verifiable ground-truth procedure and a citable corpus admits the split between procedural equivalence and evidence reconstruction, with the gap between the two indicating whether the next investment belongs in the agent or the corpus. The operating manual, script interfaces, output schema, validator, and judge-prompt structure are reproduced in the appendices with no cases and no article text.

The authors scope the study as a focused single-deployment case study, with all evidence from one internal benchmark on one corpus, make no cross-domain superiority claim, and leave a faithful port to a public procedural-support benchmark as future work. Quality is scored by LLM judges with majority voting over three runs, with estimated residual uncertainty of 2 to 3 points, and human re-scoring and a second judge family are left to future work. The head-to-head compares two end-to-end operating points rather than two retrievers at matched query and fetch effort, and the retained artifacts cannot establish whether the evidence gap survives at matched navigation effort. In the ablations, every arm's Wilson interval overlaps the full system's, and exact paired McNemar tests separate no arm at the 0.05 level, the smallest being 0.078 for removing merged entities. The inspection of the 22 cases failing evidence sufficiency was qualitative and performed by the authors, not a multi-annotator annotation with an agreement statistic, so it is reported as a supporting observation.