# JudgeBench

Using nested grouped cross-validation on 534 JudgeBench pairs and 1,802 MT-Bench comparisons, this work trains L2-regularized linear probes on residual stream activations recorded immediately before the verdict to predict position flips, where swapping candidate order changes the judge's selected response identity; AUROCs reach .621–.850 across four judges, exceeding a combined baseline of verbalized confidence, verdict logits, length differences, and the initial choice by .062–.113, and frozen probes transfer to MT-Bench at .685–.853.

Using nested grouped cross-validation on 534 JudgeBench pairs and 1,802 MT-Bench comparisons, this work trains L2-regularized linear probes on residual stream activations recorded immediately before the verdict to predict position flips, where swapping candidate order changes the judge's selected response identity; AUROCs reach .621–.850 across four judges, exceeding a combined baseline of verbalized confidence, verdict logits, length differences, and the initial choice by .062–.113, and frozen probes transfer to MT-Bench at .685–.853.

The study compares a decision-only judge, JEV, which returns a verdict plus label probabilities, against sixteen generative and reward-model judges on preference, factuality, and answer-adjudication tasks with blinded human adjudication, finding it within three percentage points of the strongest comparator, GPT-6, on ordinary preference and evidence-grounded factuality at about 0.36% of that comparator's fee, with the gap concentrated in low-confidence decisions, so a frozen cascade that accepts confident verdicts and escalates uncertain ones retains about 99% of the comparator's accuracy at lower cost.

The study compares a decision-only judge, JEV, which returns a verdict plus label probabilities, against sixteen generative and reward-model judges on preference, factuality, and answer-adjudication tasks with blinded human adjudication, finding it within three percentage points of the strongest comparator, GPT-6, on ordinary preference and evidence-grounded factuality at about 0.36% of that comparator's fee, with the gap concentrated in low-confidence decisions, so a frozen cascade that accepts confident verdicts and escalates uncertain ones retains about 99% of the comparator's accuracy at lower cost.