BayesJudge models conflicting human and LLM judgments as posterior distributions, raising expert prediction accuracy to 65.3% on SummEval
The authors propose BayesJudge, an online Bayesian meta-evaluation layer for conflicting human-LLM judgment streams that outputs a posterior verdict distribution per comparison while estimating rater-specific human confusion matrices and LLM presentation-order bias, using tie-open labels and paired AB/BA order swaps as protocol-level identification instruments; synthetic experiments recover prespecified evaluator parameters, and on SummEval it detects LLM position effects, separates expert from crowdworker behavior signatures without rater metadata, and yields posterior uncertainty that correlates with human disagreement.