Mixed-quality judge pools can still be aggregated: AGORA learns trust weights with item-response theory and holds its edge down to a few hundred items
LLM judges are a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias, but judge pools mix very different reliabilities while the human-labeled sets used to validate them hold a few hundred items or fewer. AGORA places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned jointly from all items without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items.