Mixed-quality judge pools can still be aggregated: AGORA learns trust weights with item-response theory and holds its edge down to a few hundred items
Synopsis
LLM judges are a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias, but judge pools mix very different reliabilities while the human-labeled sets used to validate them hold a few hundred items or fewer. AGORA places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned jointly from all items without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items.
Interpretation
Multi-judge aggregation no longer averages strong and weak judges at the same weight; each judge's reliability is estimated jointly from all items. The prior approach took a per-item mean, which weights weak judges as heavily as strong ones; confusion-table models such as Dawid-Skene instead estimate a per-judge error table, which needs more items than a few-hundred-item validation set provides. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items.
The aggregator places each item on one latent quality scale, and a judge's reliability is captured by how sensitive it is to item quality plus a set of ordered rating thresholds. That structure comes from item-response theory, the family of statistical models that explains ratings through item difficulty and rater ability together; here item positions and judge parameters are estimated in one model. AGORA remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and the learned trust weights track per-judge human validity.
Perspective
The demonstrated scope is an aggregation comparison on three benchmarks, covering large evaluation sets down to a few hundred items and increasingly uneven judge pools; tasks or judge types beyond these benchmarks are not described.
The material reports its conclusions in summary form, without naming the benchmarks, item counts or metric values; the size of the widening advantage as sets shrink is also not quantified.