Skip the per-retriever trial run: one calibration forecasts 14 RAG configurations and cuts diagnostic calls by 80%
Synopsis
Retrieval-augmented generation does not improve every task, since irrelevant evidence can distract a generator, and measuring each candidate retriever's gain directly repeats generation for every configuration. EviCast splits a retriever's net utility into the generator's response to controlled evidence states and that retriever's offline distribution over those states; the response is measured once on a small calibration set and reused. On 14 HotpotQA retrieval configurations, one calibration used 96 generation calls, made 14 of 14 correct deployment decisions with a 2.75 F1-point forecast error, and cut diagnostic generation calls by 80% relative to per-configuration direct measurement.
Interpretation
Retrieval value can be forecast before deployment instead of running every candidate retriever end to end. EviCast returns the expected net gain of a generator, task distribution and retriever combination over the no-retrieval baseline, and decides whether to enable retrieval. Earlier retrieval-decision methods trigger retrieval per query or generation step, evaluate a system after execution, or estimate the local utility of a query or document; none determines before large-scale execution whether a whole configuration is worth deploying or compares several candidate retrievers under a shared generation budget. Across 10 task configurations spanning case-based emotion reasoning, missing-modality augmentation, label evidence and dense passage retrieval, frozen decisions agreed with independent test conclusions in 9 of 10 cases, with no configuration lacking a significant gain falsely enabled.
One generator response calibration serves many retrievers; adding a candidate needs only offline counting of its evidence-state distribution, not new generation. Direct per-configuration measurement generates both retrieval and no-retrieval answers for each retriever, so generation cost grows linearly with the number of candidates and small-sample gain estimates vary. The 14 HotpotQA retrieval configurations shared one EviCast-B calibration: 96 generation calls, 14 of 14 correct decisions, mean absolute error of 2.75 F1 points, and Pearson and Spearman correlations of 0.983 and 0.890; against Direct, forecast error fell 71.3% and generation calls fell 80%.
In multihop question answering, evidence sets holding one correct passage can differ sharply in value, so evidence states must separate terminal from bridge passages. Counting correct passages cannot distinguish a state that supplies only bridge information from one that directly contains answer information, although both hold one annotated supporting passage. Those two states gave F1 gains of 4.55 and 50.95 points for Qwen and 2.58 and 45.65 points for Phi-4; the topology-aware EviCast-T decided 16 of 16 configurations correctly on MuSiQue and HotpotQA with an overall mean absolute error of 2.81 points, and 23 of 24 after prospective transfer to 2Wiki.
Perspective
The demonstrated setting is deployment diagnosis under a fixed generator, prompt template, corpus and retrieval depth, with only offline rankings varying; evaluation covers three multihop question-answering datasets, multimodal and text-only generators, and sparse, dense and fusion retrievers. The authors propose extending evidence states to variable retrieval depths, iterative retrieval and multimodal knowledge environments.
State sufficiency assumes the response within a state does not vary systematically with the retriever, and stratified permutation tests detected no such dependence in all 15 tests; the topology-aware map deviates from a finer reference by 0.31 and 0.27 F1 points, with remaining error attributed to calibration sampling. Exactness of the binary instantiation depends on a linearity gap that is small on the reuse split but larger for both generators on the topology-aware split.