Search pipelines can run clean and still miss the point: WISE traces the bad page back to ranking and recall
Synopsis
Industrial search pipelines can return results on time and still show pages that fail the user's intent, while existing monitoring watches latency, availability and errors. WISE splits the diagnosis in two: it first judges whether a page has a quality problem using only user-facing multimodal evidence, then pulls internal pipeline evidence to attribute confirmed problems to supply, recall, relevance scoring, ranking or reranking. On WISE-Bench, 12,000 real search cases, Qwen3.5-35B-A3B rises from 35.25% to 75.20% end-to-end exact match after trajectory-supervised fine-tuning and AHPO.
Interpretation
Search quality diagnosis moves from judging whether results are good to naming which pipeline stage produced the problem, placing user-facing experience evidence alongside internal candidate supply, recall paths, model scores and stage-wise ranking positions in one diagnosis. Earlier search benchmarks mainly measure user-facing result quality and lack internal pipeline evidence and attribution labels, while existing root-cause analysis targets operational failures such as latency and errors rather than the quality of algorithmic outputs. WISE-Bench draws 12,000 real cases randomly sampled from Kuaishou's main search scenario between July 1 and July 10; each case carries the query, request time, scenario, top-10 displayed results, multimodal experience evidence, pipeline traces and expert diagnosis annotations, split into 8,000 training, 2,000 validation and 2,000 test cases.
The diagnostic agent first finds experience problems on the page, then investigates internal causes only for confirmed problems, with the two stages deliberately kept on separate evidence. Problem discovery may use only the query and displayed content, including text, speech recognition, optical character recognition, visuals, publication time and author or source; internal pipeline evidence enters the attribution stage only after an issue is confirmed. The execution harness controls the context and tool permissions each agent can see by role, with separate roles for problem discovery, attribution and report generation, and structured records passed between nodes and checked for schema compliance and evidence references.
Training data comes from minimally repairing a teacher model's execution trajectories rather than regenerating whole trajectories or correcting only the final answer. Expert annotations give diagnostic outcomes but not the intermediate decisions and tool calls an agent needs, and fine-tuning directly on raw trajectories would teach the model their incorrect decisions. ReTrace keeps every decision, tool call and observation before the earliest repairable decision and repairs only the affected suffix, accepting a repair only if the final diagnosis matches the expert annotation and each decision is supported by evidence available at that step; against full regeneration it matches 57.87% case attribution exact match while cutting teacher generation tokens from 32.0k to 20.4k.
Once a failure is localized to the earliest evidence-supported wrong decision, the system decides whether to revise the execution harness or the model policy and applies that update. A diagnostic failure can come from the model deciding wrongly when evidence is sufficient, or from the harness failing to expose needed evidence, actions or state expressiveness, so the two failure modes need different optimization surfaces. On 200 manually adjudicated development-set failure instances, direct routing reaches 58.9% accuracy and 43.6% macro-F1, adding the failure analyzer raises these to 69.2% and 60.0%, and the full AHPO router reaches 84.2% and 84.1%; on Qwen3.5-35B-A3B, supervised fine-tuning lifts end-to-end exact match from 35.25% to 69.80% and AHPO to 75.20%.
Perspective
The demonstrated scope is whole-page multimodal search diagnosis in the main search scenario of Kuaishou's large-scale short-video platform, with benchmark cases sampled from requests between July 1 and July 10 and evaluation focused on case-level problem detection, item-level localization and pipeline attribution; standalone page-level diversity issues are rare in the sampled traffic, so evaluation does not center on them.
Failure localization is explicitly described as a trace-based account of error propagation rather than a counterfactual causal claim; intermediate development-set gains do not always translate into test-set improvements; item accuracy on Qwen3.5-4B dips slightly from 83.91% to 83.48%; and when evidence cannot distinguish a primary component, attribution is marked unresolved while the confirmed experience issue is kept.