A fixed exam for Japanese search agents: 250 questions over a frozen corpus, and Qwen3.8-27B answers 89.2% with tools
Synopsis
Japanese search agents previously lacked a repeatable quantitative evaluation, and most Japanese benchmarks needing retrieval asked for only a single lookup. Preferred Networks built and released Snapshot BrowseComp-ja, which poses 250 Japanese questions against a frozen web corpus of about 8.24 million Japanese and 45.52 million English documents. All three public models scored low without a retrieval tool, and Qwen3.8-27B reached 89.2% on all 250 questions with one.
Interpretation
Japanese search agents now have a repeatable benchmark whose questions require repeated retrieval over a fixed web corpus and combining several pieces of information into a Japanese answer. Japanese lacked a quantitative benchmark aimed at search agents, because Japanese tasks that need retrieval were mostly single-lookup RAG-style questions, which do not test an agent that searches in several steps. The benchmark holds 250 questions, 200 set at lite difficulty and 50 at hard difficulty; without a retrieval tool the three public models scored 5.6% for Gemma-4-31B-it, 7.2% for Qwen3.8-27B and 21.6% for GLM-5.3-Flash, indicating that most questions do depend on external retrieval.
The retrieval target is a frozen web-text corpus rather than a live web search service, so scores from the same model at different times are directly comparable. Calling real web search makes results shift over time and incurs API costs, so frequent evaluation is hard to keep under identical conditions. The corpus draws on FineWiki and FineWeb-series data, with about 45.52 million English documents and about 8.24 million Japanese documents; questions were generated automatically by an agent searching that corpus, accepted only when the answer follows from the corpus, no alternative answer exists in it, and the set difficulty is met.
Opening a retrieval tool raises accuracy sharply, and the hard subset spreads strong models into a comparable range. Without tools the higher score belongs to the larger GLM-5.3-Flash, while the roughly 30B Gemma-4-31B-it and Qwen3.8-27B sit close together, broadly reflecting how much knowledge the parameters hold. With tools Qwen3.8-27B scored 89.2% on all 250 questions (94.5% lite, 68.0% hard), GLM-5.3-Flash 87.2% (94.5% lite, 58.0% hard) and Gemma-4-31B-it 44.8%; on lite the two strong models nearly saturate, while the 50 hard questions land between 40% and 70%.
Perspective
The benchmark targets development-time evaluation of Japanese search agents, retrieves from a frozen web-text corpus, and splits questions into lite and hard tiers for comparing models of different scales.
A frozen corpus cannot fully reproduce agent capability under real web search, and existing work reports problems caused by fixed corpora, which this benchmark may share; questions are model-generated, so their quality and answer uniqueness need continued verification; Qwen3.8-27B and GLM-5.3-Flash perform nearly the same with tools, and whether the question format disadvantages GLM-5.3-Flash or the two differ little as search agents is currently unclear.