A finite-sample certificate for which fragments a retrieval corpus should keep, valid even after data-dependent selection
Synopsis
When a retrieval-augmented generation corpus must be trimmed under a storage or annotation budget, how much query coverage the chosen fragments lose relative to the best feasible set has lacked a finite-sample guarantee that survives data-dependent selection. This work separates selection from auditing: a pilot picks the set, screens competitors and builds a comparison network, then an independent audit acquires relevance labels with probabilities constant within observable strata and yields a simultaneous bound on all retained coverage contrasts, with simulations confirming the nominal level.
Interpretation
Budget-constrained selection of documentary fragments can now carry a finite-sample, distribution-free upper bound on coverage regret that remains valid after the set and its competitors are chosen from data. Previously, a near-optimal value on the empirical selection objective did not imply small population regret, so a computational approximation guarantee and a statistical coverage certificate were distinct things. Across all settings and all seven certificates, the simulation's certificate covered the true regret in every replication, and the simultaneous audit event held in every replication, so the certificates are valid with room to spare from conservative constants.
The construction splits selection from labeling: a pilot stage may freely choose the set, screen competitors, build comparison networks and learn an annotation policy, and then an independent audit acquires relevance labels with probabilities that are constant within observable strata. Because acquisition probabilities stay constant inside each observable stratum, adaptive sampling does not invalidate the confidence statement about the unknown query population. The bound becomes observable through empirical-Bernstein variance calibration or a one-stage construction without calibration, and it accounts for estimated stratum proportions; the simulation runs on one synthetic design with six clusters of five fragments, unit costs and a budget of 5.
Perspective
Validity holds under the assumptions stated in the text, including independent unlabelled queries, a stratified audit design that is label-ignorable, and pre-audit residual predictors; the simulation runs on one synthetic design with six clusters of five fragments, unit costs and a budget of 5.
The text separates validity, established under specified assumptions, from efficiency, which depends on the geometry of the competitors and on the audit design, and notes that the multiscale network's improvement over a direct union bound requires a certified redundancy envelope that samples alone cannot certify; in the simulation the certificates stay far above the true regret, with the smallest at about 0.101, of which about 0.033 is the weight-correction term.