Judges agree more, humans no closer: rewriting only contested rubric clauses lifts panel consensus

Disagreement among judges is no longer averaged away; it is treated as a signal of where a rubric admits more than one reading, and used to locate and rewrite the contested clauses. The prior practice averages panel scores or rewrites the whole rubric once; here only the selected contested clauses are edited and the rest of the rubric text is left unchanged. On SummEval, editing contested clauses raised Krippendorff's alpha across five judges from 0.577 to 0.646, 0.069 above the original rubric (95% CI 0.046 to 0.092) and 0.074 above a control that edited 23 randomly chosen clauses (0.047 to 0.097).

The procedure first splits each rubric into atomic sub-criteria, then ranks them by how often judge pairs give different verdicts, and rewrites only the few clauses with the highest disagreement rate. Selection is deterministic: each dimension takes k equal to min(5, n/2) sub-criteria with a positive disagreement rate, ordered by decreasing rate and by identifier on ties, and the editor model sees only the clause, the items where judges split, and the judges' rationales. The evaluation covers SummEval, the HealthBench meta-evaluation, and USR Topical-Chat, three benchmarks with human labels, with five judges from three providers and 129,770 judge requests; each of the two edited conditions changed 23 clauses.

Judges agreeing with each other more does not mean they moved closer to human annotators. On SummEval, agreement with expert ratings changed by -0.007 after editing contested clauses (95% CI -0.037 to 0.021), with no detectable change, while the random-clause control reached the highest ensemble human agreement at 0.516. On HealthBench and USR neither inter-judge agreement nor human agreement met the preregistered effect criteria, and in the preregistered secondary comparison human agreement after editing contested clauses was lower than after editing control clauses in point estimate on all three datasets.

The procedure targets evaluation settings that use a panel of judges to score open-ended output; it takes a rubric, a development split, and a panel, and returns an edited rubric, reading no human labels, which are used only to evaluate the outcome. On SummEval, editing contested clauses met the preregistered effect criterion against both the original rubric and the random-clause control. On HealthBench only 5 of 34 criteria were edited, covering 120 of 799 evaluation units, 15 percent of them; on USR several dimensions already had high development agreement and the dataset is small. The code-transfer stage was not exercised: the editor mapped none of the 75 sub-criteria to the four deterministic checkers, so the 0.9 transfer threshold was never tested.

Three of the five judges are Claude models and the other two providers are represented only by small models, while reasoning settings and output caps differ across judges, so provider, model size, and reasoning effects are confounded. The editor model belongs to the same family as three judges. The evaluation uses existing human labels, which are themselves noisy, and three English datasets; SummEval is widely used and may appear in judges' training data. No equivalence tests were run, so the absence of a detectable change should not be read as evidence of no change, and the secondary comparison's intervals include zero on all three datasets.