Web-search agent evaluation keeps the real trail: flagged failure nodes link back to the answer groups that went wrong
Synopsis
Evaluations of long-horizon web-search agents have leaned on simulated trajectories or on trails reconstructed from final answers, which makes the search process itself hard to inspect. LongSearchEval runs 16 models on 100 tasks under one shared agent controller and five restricted web tools, retains 1,600 real trajectories, verifies answer groups against the webpages the agent itself cited, then links flagged failure nodes to groups judged incorrect. Human review agrees with 94.50% of automatic judgments on 946 answer groups, and 311 of 317 sampled round-to-group links pass a feature-correspondence check.
Interpretation
The object of web-search agent evaluation moves from the final answer to the answer plus a real interaction trail. Every evaluated model runs under the same agent controller, prompt and five web tools, and each search, page opening, reading, in-page lookup and link extraction is kept in its original order. Earlier evaluations mainly relied on simulated trajectories or on trails reconstructed after the fact from final answers rather than retained real interaction records, which limited behavioral fidelity and weakened fine-grained error localization. Sixteen models from eight families each contribute one retained run on 100 long-horizon web-search tasks, giving 1,600 trajectories; collection and evaluation took 1,037 and 318 cumulative hours.
Evaluation proceeds in two steps: first judge each answer group against only the webpages the agent cited, then trace groups judged incorrect back to related rounds in the trail. The first step needs no human-written gold answer and fetches only URLs given in the response; missing citations, unsupported claims, invalid attributions and constraint violations count as incorrect, while groups that fail only through timeouts, network failures or anti-bot blocking are kept separately as unverifiable. Two judge models assess each round independently and disagreements go to an arbiter; of 42,658 normalized rounds, 14,734 evaluable records are retained, including 6,545 automatically flagged candidate failure nodes.
Flagged failure nodes can be linked back to specific incorrect answer groups, producing evidence locations that can be revisited. Of the 6,545 candidate nodes, 3,664 retain links to groups judged incorrect in the first step, covering 2,229 of 3,374 incorrect groups; these locations mark where related content appears, not when the agent first made an erroneous commitment. Among 850 runs with at least one link, the earliest association appears by round 5 in 46.5% and by round 20 in 88.7%, with a median of round 6; human review of 317 retained links confirms 311 locatable feature correspondences, a precision of 98.11%.
Perspective
The evaluation covers 100 public-web tasks that require multi-source search and constraint satisfaction, with one retained run per model-task cell under a shared weak-tool interface; reported intervals describe task-level uncertainty rather than variation across repeated agent executions. Visual interaction, authenticated workflows, private context and transactional operations form a separate setting.
Localization is a post-hoc association conditioned on the final answer and the first-stage judgments, and does not identify when the agent first made an erroneous commitment; the earliest association only marks the earliest round in which returned related content appears. First-stage judgment errors can affect retained associations, and 1,145 of the 3,374 incorrect answer groups still have no retained content association.