Skip to main content
Search

# Web Agent Evaluation

2 related articles

Subscribe to updates
  1. arXiv

    Web-search agent evaluation keeps the real trail: flagged failure nodes link back to the answer groups that went wrong

    Evaluations of long-horizon web-search agents have leaned on simulated trajectories or on trails reconstructed from final answers, which makes the search process itself hard to inspect. LongSearchEval runs 16 models on 100 tasks under one shared agent controller and five restricted web tools, retains 1,600 real trajectories, verifies answer groups against the webpages the agent itself cited, then links flagged failure nodes to groups judged incorrect. Human review agrees with 94.50% of automatic judgments on 946 answer groups, and 311 of 317 sampled round-to-group links pass a feature-correspondence check.

    Northeastern University
    Jialiang Liu（First author）
    LongSearchEvalSource-Constrained Outcome VerificationFailure-Node LocalizationWeb Agent Evaluation
  2. arXiv

    WebArxiv evaluates web agents on 510 static-snapshot tasks, finds reliance on fixed interaction histories, and adds a dynamic-memory mechanism

    The work introduces WebArxiv, a static-snapshot benchmark built on arXiv with 510 time-invariant tasks, each with a unique deterministic ground truth, for evaluating multimodal web agents on scholarly tasks such as multi-constraint paper retrieval, fine-grained content extraction, and cross-paper comparison; evaluations of a range of foundation-model-based web agents show the benchmark remains challenging, behavioral analysis reveals that agents over-rely on fixed interaction histories, causing incomplete or repetitive reasoning, and the authors therefore equip agents with a lightweight dynamic-memory mechanism for adaptive retrieval and reasoning over relevant context.

    Zihao Sun（First author）
    WebArxivDynamic-Memory MechanismWeb Agent Evaluation

You’re all caught up