Not every document question needs an agent: RouteDRAG sends 79.3% of queries to agentic execution and gains 3.47 points on average
Synopsis
In multimodal long-document question answering, agentic retrieval is not always worth its extra computation and can even hurt answer accuracy. RouteDRAG makes both whether to invoke an agent and how it then executes into learned choices: a graph-backed index organizes cross-page, table and figure content, path routing predicts the benefit of an agent over shorter paths, and dynamic execution controls tool use and termination. Across two benchmarks it assigns only 79.3% of queries to an agent while averaging 3.47 percentage points higher accuracy than shorter-path-only execution, and it also surpasses uniform agent execution with fewer answering calls.
Interpretation
Multimodal document question answering now decides per query whether to run an agent, instead of sending every question through a retrieval-and-reasoning loop. Prior agentic retrieval-augmented generation ran iterative retrieval and reasoning for each query, yet such execution is not always necessary, adding computation and potentially producing incorrect answers. An empirical analysis of queries shows that the benefit of agentic execution is query-dependent, and that additional operations may improve, preserve, or degrade answer accuracy.
The system first judges whether an agent is worth it relative to a shorter execution path, then constrains tool use and stopping inside the agent. RouteDRAG combines a unified document index, adaptive path routing, dynamic agent execution and sequential policy learning; path routing predicts the relative benefit of an agent over shorter paths, and dynamic execution maintains an evolving workspace and refines planner proposals. Sequential policy learning first trains operation control from observed outcomes and then trains path routing using the resulting agent's performance, with evaluation on two benchmarks.
Saving agent calls does not cost accuracy: the system beats both comparison settings on both benchmarks. The comparisons are shorter-path-only execution and uniform agent execution. RouteDRAG averages 3.47 percentage points higher accuracy than shorter-path-only execution across two benchmarks, assigns only 79.3% of queries to an agent, and surpasses uniform agent execution on both benchmarks while requiring fewer answering calls.
Perspective
The demonstrated scope is average accuracy, agent query allocation and answering-call comparisons on two multimodal document QA benchmarks; code is released.
The summary reports only averaged gains over two benchmarks and does not give per-benchmark values; the empirical analysis of query-dependent agent benefit is described only at summary level.