LLM labels stop being ground truth: AAELI spends human audits where they matter, reaching the same precision with about 21% fewer labels
Synopsis
Large language models can label every instance cheaply, but their judgments are biased and their errors correlated, so treating them as human labels can mislead statistical inference. AAELI uses model outputs as auxiliary information to guide a limited human audit and builds anytime-valid confidence sequences from predictable empirical-likelihood processes, so coverage survives data-dependent stopping. On Chatbot Arena, AAELI needs about 870 human labels on average to reach a confidence-sequence width of 0.10, versus about 1,100 on average for the strongest baseline, RASI, a saving of roughly 21%.
Interpretation
AAELI lets LLM judgments guide human auditing and supply auxiliary moments instead of standing in for human labels, so the population mean of human labels can be estimated under a limited audit budget. Prediction-powered inference combined model predictions with a small set of human labels to improve estimation but did not use predictions to decide which instances receive human labels; active statistical inference allocated labels by predictive uncertainty but took the predictor as given. On Chatbot Arena, AAELI needs about 870 human labels on average to reach a confidence-sequence width of 0.10, versus about 1,100 on average for the strongest baseline, RASI, a saving of roughly 21%; on AllSides and Misinfo it also reaches target widths with fewer human labels.
The method stacks inverse-probability-weighted human labels with auxiliary moments built from model outputs into augmented estimating equations, and empirical likelihood then sets the combination coefficient automatically. Empirical likelihood has long incorporated auxiliary information through moment conditions, but existing treatments assume labeled and unlabeled samples are given, with i.i.d. labeled data collected in advance, and do not consider how instances are selected for labeling. Under adaptive auditing these estimating equations have conditional mean zero at the true population mean and form a martingale difference sequence, so no unbiasedness, calibration or independence assumption on the LLM judges is required.
Human audit probabilities are allocated in proportion to the conditional standard deviation of the human label, learned online from previously audited data, approaching the oracle variance under the same audit-probability floor and long-run budget. Allocating measurement effort in proportion to conditional standard deviations is classical in survey sampling, but the attainable efficiency had not been characterized jointly for auxiliary moments built from LLM outputs and the audit rule. On synthetic data the learned audit rule nearly matches the oracle rule using known truth and saves 10.4% of human labels relative to uniform auditing at width 0.10, whereas auditing by judge uncertainty performs similarly to uniform auditing and auditing by judge disagreement is markedly worse.
Perspective
The demonstrated scope is estimation of the population mean of human labels, covering synthetic data and three real datasets, Chatbot Arena, AllSides and Misinfo, with three low-cost LLM judges per instance as auxiliary information in the real-data applications. It targets researchers and engineering teams that monitor precision in a streaming annotation pipeline and stop once a target is reached.
The analysis assumes a stationary evaluation population and noise-free human labels; distribution shift over time and noisy or inconsistent human annotations are not handled. It also treats LLM judgments as freely available for every instance and does not account for heterogeneous costs of querying multiple judges.