WebArxiv evaluates web agents on 510 static-snapshot tasks, finds reliance on fixed interaction histories, and adds a dynamic-memory mechanism

Introduces the WebArxiv benchmark: built on static arXiv snapshots, comprising 510 time-invariant tasks, each with a unique deterministic ground truth. Relative to existing benchmarks that emphasize general-purpose browsing and often depend on live sites whose changing content and structure undermine reproducibility, this work places evaluation in research-oriented scholarly discovery workflows and uses static snapshots to ensure reproducibility. The text explicitly states the task count of 510, the static-snapshot form, and the time-invariant and unique-deterministic-ground-truth design properties, and describes arXiv as a realistic, reproducible, hierarchically structured, information-centric testbed without privacy-sensitive interactions.

Task design goes beyond simple information lookup and rule following to emphasize multi-constraint paper retrieval, fine-grained content extraction, and cross-paper comparison. It brings composite constraints and cross-document comparison from scholarly discovery into the evaluation tasks, rather than staying at general browsing or single-point information location. The text supports this claim through task-type descriptions; it does not give the count distribution or difficulty tiers for each task type.

Evaluations of a range of foundation-model-based web agents show WebArxiv remains challenging; behavioral analysis finds agents over-rely on fixed interaction histories, causing incomplete or repetitive reasoning. Beyond benchmark evaluation, it adds a behavioral-level diagnosis linking performance shortfalls to a specific pattern of interaction-history reliance. The text reports evaluation results and behavioral-analysis conclusions, but does not give specific scores, a model list, or sample sizes.

Equips agents with a lightweight dynamic-memory mechanism for adaptive retrieval and reasoning over relevant context. It proposes a corresponding mechanism for the fixed-interaction-history reliance identified above, rather than stopping at the benchmark and diagnosis. The text describes the mechanism as lightweight and aimed at adaptive retrieval and reasoning, but does not report the specific magnitude of improvement it yields.

The benchmark targets research-oriented scholarly discovery workflows and is meant for evaluating multimodal web agents in static-snapshot environments such as arXiv that are hierarchically structured, information-centric, and free of privacy-sensitive interactions; its time-invariant tasks and unique deterministic ground truth let results be reproduced and compared without drifting as live sites change. The dynamic-memory mechanism targets agents that need adaptive retrieval and reasoning, for researchers to continue experimenting on similar scholarly retrieval and cross-paper comparison tasks.

The text is abstract-level material: it does not give the count distribution across task types, the list of evaluated models, specific scores, the magnitude of improvement from the dynamic-memory mechanism, or how ground truth is constructed and validated, so readers still need the original to confirm these details. Whether the dynamic-memory mechanism works equally well on other sites or non-scholarly tasks is also not addressed and remains an open question to watch.