Disagreement labels alone can't settle an F1 ranking: two agreement counts give exact bounds, and a 0.01 tolerance cuts labeling to 74.59
Synopsis
After labeling only the items where two classifiers disagree, two agreement counts yield exact bounds on the F1 difference, enough to certify which classifier to pick within a stated tolerance or to say which agreement labels still need querying. Across 3,200 frozen model pairs, bounds that share one label completion certified 79.46% of choices in the reserved cells at a 0.01 tolerance, versus 6.57% for separate bounds; under magnitude ordering, the 0.01 tolerance cut total labels for all pairs from 121.16 to 74.59, against 500 for full labeling. The guarantee covers the evaluated batch, not future data.
Interpretation
Revealing labels only on items where two classifiers disagree does not always determine which has the higher F1; once the unknown agreement labels are reduced to two counts, the F1 difference is pinned inside an exact interval, which either certifies a choice at the requested tolerance or calls for more agreement queries. Earlier routes either assume baseline sensitivity and specificity that transfer to the validation data, or pursue label efficiency without a deterministic F1 guarantee over every label completion; the accuracy difference is already fixed once disagreement labels are revealed, while the F1 ranking can still flip with the allocation of the remaining labels. In the 28 reserved cells, 20.54% of pairs on average carry a compatible opposite advantage of at least 0.01 without prevalence information, and 10.52% with exact prevalence; no pair sits on the 0.01 boundary, so complements are certifiable at that tolerance.
How tight the interval is depends on keeping the label dependence between the two classifiers on the same items, rather than bounding each one separately. Bounding each model separately and then comparing discards the constraint that both share one set of unknown labels, and the resulting intervals are markedly wider. In the reserved cells at a 0.01 tolerance, joint bounds certify 79.46% of choices against 6.57% for separate bounds; with exact prevalence the figures are 89.48% and 18.80%, with mean widths of 0.173 versus 1.585 and 0.034 versus 1.005.
Ordering agreement queries by predicted-count imbalance, together with a 0.01 tolerance, sharply reduces how many items must be labeled. Requiring a strict winner or an exact tie costs more labels on the same pairs, and full labeling costs 500 labels per pair. Under magnitude ordering, total labels fall from 121.16 for strict/exact decisions to 74.59 at the 0.01 tolerance across all pairs, and from 92.57 to 40.29 on validation-selected pairs; zero-loss controls total 120.92 and 92.25, so most of the saving comes from positive tolerance rather than ties.
Perspective
The conclusions apply to an evaluated batch with fixed predictions, correct labels and binary positive-class F1; the evaluation covers 3,200 model pairs deduplicated from 36 cells, built on existing prediction artifacts from TableShift, EMBER, ToN-IoT and UNSW-NB15, with no model refitting and no new data collected.
Compatible extreme label completions need not be probable, so certification is not the same as realized error; costs are neither optimal-acquisition lower bounds nor additive across pairs; statistical alternatives can trade assumptions or error probability for fewer labels. The authors state that disagreement-first labeling is imposed rather than optional, and benchmark reuse limits breadth.