Agents stop re-deriving cross-document answers: a compiled wiki lifts accuracy from 0.10 to 0.52, and an evidence graph adds 27%
Synopsis
A memory layer for agents doing cross-document reasoning now has a controlled comparison: on 111 synthetic insurance documents, an agent using the OpenWiki-compiled wiki reached 0.52 accuracy on 50 questions requiring connections across sources, against 0.10 with keyword search alone; adding an evidence graph built from citations and document references raised accuracy a further 27%, doubled citation consistency and cut cost per correct answer by 58%. On simple lookups the two conditions were about equal, so the wiki brought no advantage there.
Interpretation
For cross-document questions, an agent can treat a compiled wiki as a reusable memory layer instead of rebuilding the relationships between concepts from raw documents on every question. Retrieval pipelines previously used semantic or hybrid search to fetch isolated chunks, leaving the agent to reconstruct how chunks relate and which rule supersedes which. On 50 questions requiring dependency tracing across 111 synthetic insurance documents, the agent with the OpenWiki-compiled wiki scored 0.52 accuracy against 0.10 with keyword search alone, about 5x, and cost per correct answer fell from about $0.55 to $0.19.
An evidence graph layered on the wiki lets the agent walk explicit edges from a statement to its supporting passage, making cited evidence more stable. With the wiki alone, citation overlap on connected questions dropped to 0.25 from 0.74 on simple lookups, so the same question cited different sources across three runs. Compared with the wiki alone, adding the evidence graph raised accuracy 27%, doubled citation consistency, cut median answer time 40% and cut cost per correct answer 58%.
The evidence graph is materialized deterministically from the wiki's existing text citations and document references, so building it needs no additional model calls. Extraction-heavy knowledge graph pipelines typically use a model to extract and reconcile entities, whereas this layer reuses references already present. The graph took under a second to build and resolved every explicit cross-reference in the test library; paragraphs were given unique identifiers and became the atomic units linking claims to exact supporting passages.
The payoff from compiled memory depends on reuse, and simple lookups do not benefit. On 80 single-passage lookup questions, search and wiki scored 0.84 and 0.83 accuracy, 0.72 and 0.74 citation overlap, and $0.020 and $0.021 per correct answer respectively. In the same comparison the wiki raised latency from 9 to 12 seconds; compiling 111 documents into a wiki roughly one-tenth their size cost $4.15, later added documents cost about $0.09 each, and reported costs exclude compilation and updates.
Perspective
The evaluation covers a library of 111 synthetic insurance documents with 80 simple lookup questions and 50 connected questions, each asked three times; the comparison changes only whether the agent can access compiled memory, holding model, instructions, documents and questions constant. Reported costs exclude wiki compilation and update.
The wiki's citations are model-generated, so edges derived from the wiki depend on their accuracy; the evidence graph improved accuracy and consistency but did not make the agent fully deterministic. On simple lookups search matched the wiki while being faster, and compilation and maintenance costs may outweigh retrieval savings before reuse is high.