Geopolitical intelligence QA gets a public benchmark: 415 reports, 5,525 questions, retrieval solved but reasoning still hard

Geopolitical and strategic intelligence reports now have a reproducible question-answering benchmark spanning factual, inferential, comparative, and risk-oriented questions. Previously this document class could only be searched by hand, with no systematic evaluation built for it. The corpus holds 415 open-source institutional publications from 122 publishers, with a median publication year of 2025 and 94% dated 2022 or later; generation produced 5,525 question-answer pairs over 413 reports.

Retrieval does not need larger embedding models, and systems nearly tie at the document level. The common expectation is that semantic retrieval quality rises with embedding-model scale and that commercial APIs lead. On 4,980 clean queries over 55,353 chunks, every system reaches document-level R@10 above 0.94; at chunk level bge-m3 scores 0.785 nDCG@10 against 0.784 for the 8B Qwen3-Embedding-8B, a non-significant gap, while significantly outperforming commercial APIs such as OpenAI-3-large.

Generation is the real bottleneck: given correct evidence, answer quality still falls as reasoning demand rises. Earlier evaluations often treated retrieval failure as the main error source. On 250 human-annotated items, the strongest reader Llama-3.3-70B scores 2.56 overall, with 28% of factual answers flawed, rising to 38% comparative, 47% inferential, and 70% risk-oriented; errors include unsupported claims, incomplete or incorrect answers, and unjustified refusals when evidence is present.

The benchmark targets researchers studying retrieval and generation, using 415 open-source English institutional publications; retrieval is evaluated on 4,980 clean queries over 55,353 chunks, and readers on 250 human-annotated items. The releasing team shares derived chunks, question-answer pairs, answer-cited gold chunks, and per-item judge scores, while source PDFs are not republished and are instead attributed and linked to their publishers.

The corpus is English-language and strongly contemporary, with 94% of documents published since 2022, so it does not represent global geopolitical analysis; the question mix is dominated by factual items, reflecting document affordances and the generation pipeline. The human-annotated sample is small, inter-annotator agreement is modest, and the annotators are colleagues of the pipeline designers, so independent third-party annotation would be a stronger check. Evaluation is single-run with one judge, and gold is defined by the chunks cited by LLM-written reference answers.