Bringing survey small-area estimation into AI evaluation: PP-S and PP-TS improve point and interval estimates on a benchmark and on deployed traffic, while DB-CV picks as well as an independent validation sample from one sample

It introduces prediction-powered smoothing (PP-S), which takes each domain's GREG (generalized regression) estimate as input to a Bayesian Fay–Herriot-type small-area model that borrows strength across domains; its extension PP-TS replaces the single random effect with a sum of effects along the levels of a nested reporting taxonomy. Direct estimators such as PPI and GREG use only a domain's own labels and stay imprecise where labels are few, with variance scaling as 1/n; small-area estimation had mainly served official survey statistics, and this work integrates it with prediction-powered inference into an estimation workflow for AI evaluation, letting auxiliary information enter both at the unit level (GREG) and at the domain level (regression covariates). On the Open LLM Leaderboard (9,324 questions, 34 task types, a two-level taxonomy over five benchmarks), PP-TS at a 10% budget has lower RMSE and interval score than every direct and smoothed estimator under all three auxiliaries; on PRISM traffic (68,371 ratings, 63 domains), PP-S with judge plus content covariates lowers oracle RMSE from 2.505 for HT to 1.527 and interval width from 10.47 to 5.59.

It derives DB-CV, an approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators, which removes the sample-level bias left after the fold-level correction of Dong and Li (2026). Cross-validation in AI evaluation had been used to tune an estimator rather than to choose among them, and information criteria require a likelihood and cannot score direct estimators; DB-CV lets a single sample support comparisons across estimator classes and yields a calibrated report of the chosen estimator's error. On PRISM over 100 replications at a 10% budget, DB-CV's chosen candidate has oracle RMSE 1.543, rank correlation 0.89, and reported-to-oracle RMSE ratio 1.07, versus 1.528/0.88/4.00 for naive CV, 1.557/0.78/2.59 for a 50/50 split, and 1.678/0.62/3.54 for an 80/20 split; at a 20% budget the conclusions carry over (DB-CV ratio 1.11).

Using PRISM, it builds a nonprobability-sampling contrast showing what happens when the selection mechanism is ignored: sampling rejected responses at about twice the rate of chosen ones and then estimating as if at random yields a sample-mean bias matching a typical online opt-in panel, a bias that persists as the budget grows while 95% interval coverage falls well below nominal. It makes Meng's (2018) big-data paradox concrete for AI evaluation and notes that PPI does not fix the problem either, because without accounting for selection the residuals from the labeled sample remain biased for the population residuals. The demonstration uses PRISM participant ratings of every response (chosen responses average 81.7 versus 54.1 for the rest), with sampling probability proportional to a standardized chosen indicator, putting the correlation between being sampled and the rating at about 0.03 in magnitude at the 5% budget; the authors therefore treat a probability sample as the foundation for the rest of the paper.

In the benchmark study it decomposes where PP-TS's gain comes from: pooling alone (HT input, intercept-only linking) barely improves on HT, while most of the gain comes from the domain-level covariate and from taxonomy smoothing, and an auxiliary's value at the domain level cannot be inferred from its unit-level accuracy. This addresses the practical question of which smoothing model to use and how much smoothing to introduce, previously identified as an open question, and shows that two weak auxiliaries with nearly identical unit-level correlations differ substantially in the RMSE reduction their domain means deliver. At a 10% budget the strong auxiliary, historical difficulty (unit-level correlation 0.61), gives PP-TS an RMSE of 0.059; the two weak auxiliaries correlate 0.31 and 0.33 at the unit level, yet the generalist's domain mean lowers RMSE far more than the specialist's, and taxonomy smoothing absorbs the specialist's uneven error pattern across MATH and IFEval.

The workflow targets disaggregated evaluation built on a probability sample, where reporting dimensions partition the population into domains of known size, and it applies to both benchmark task types and deployed system traffic; when domain-level covariates are available, the smoothing models can also estimate a domain with no labeled units, though the authors list the reliability of such zero-shot domain evaluation as an open question. For practitioners, what is directly reusable is the three-step workflow of designing the sample, fitting candidate estimators, and using DB-CV on a single sample to select and report error, together with the public code and processed data.

Each setting is covered by a single dataset: the benchmark side is phi-4's 34 task types from the Open LLM Leaderboard, and the traffic side is 63 domains from PRISM, so behavior across more models, more reporting dimensions, and different sampling designs remains to be seen. The authors also note that the two assumptions behind the smoothing models, the sampling-model approximation for the direct estimate and the specification of the linking model, matter most in the smallest domains, and that how much an auxiliary helps at the domain level and how much taxonomy smoothing adds both vary with the data, so the linking structure must be validated. In addition, DB-CV's sample-level bias correction relies on the approximation that each leave-one-fold fit centers on the full-sample fit up to a higher-order remainder; the authors report that the adjusted score of Dong and Li (2026) abstains on every pairwise comparison in this task because of its conservative bounds, indicating that different bias treatments still behave differently in practice. Open questions include the conditions under which zero-shot domain evaluation is reliable, carrying grader disagreement into the uncertainty of an estimate as measurement error, and combining the large nonprobability samples deployed systems generate with a probability sample to guard against selection bias.