# Inter-Judge Agreement

2 related articles

arXiv Oct 14, 2026
Customer-support agent benchmark scores 17 models with five judges: GPT-5.4 wins everywhere, judges agree only moderately
A 500-scenario zero-shot benchmark built from real customer-support transcripts scores 17 models with five independent judges, and GPT-5.4 ranks first under every judge while judges agree only moderately on pass/fail.
Md Tahmid Rahman Laskar（First author）
Skill Execution Agent evaluation LLM-as-a-Judge Inter-Judge Agreement
arXiv Oct 12, 2026
Judges agree more, humans no closer: rewriting only contested rubric clauses lifts panel consensus
A label-free procedure turns disagreement among LLM judges into rubric repairs, rewriting only the most contested clauses; on SummEval it raised inter-judge agreement by 0.069 while agreement with expert ratings showed no detectable change.
한상도（First author）
Label-Free Rubric Clarification LLM-as-a-Judge Inter-Judge Agreement Agreement with Human Annotators

You’re all caught up

Customer-support agent benchmark scores 17 models with five judges: GPT-5.4 wins everywhere, judges agree only moderately