WebArxiv evaluates web agents on 510 static-snapshot tasks, finds reliance on fixed interaction histories, and adds a dynamic-memory mechanism
The work introduces WebArxiv, a static-snapshot benchmark built on arXiv with 510 time-invariant tasks, each with a unique deterministic ground truth, for evaluating multimodal web agents on scholarly tasks such as multi-constraint paper retrieval, fine-grained content extraction, and cross-paper comparison; evaluations of a range of foundation-model-based web agents show the benchmark remains challenging, behavioral analysis reveals that agents over-rely on fixed interaction histories, causing incomplete or repetitive reasoning, and the authors therefore equip agents with a lightweight dynamic-memory mechanism for adaptive retrieval and reasoning over relevant context.