Rubric scoring stops guessing from wording alone: tree search plus response calibration tops five of six preference benchmarks

Rubric construction moves from one-shot generation to a constrained search over a hierarchy, so evaluation criteria are split into independently assessable items. Earlier methods either generated criteria directly from the task instruction or derived them by contrasting preferred and rejected responses, and guided optimization only with aggregate outcomes such as preference accuracy or downstream reward. On 1,500 instances stratified from HealthBench and RubricBench, tree-structured generation beats both flat generation and a call-matched Flat-Multi on coverage and valid novelty under two independent judge models.

Whether a criterion works cannot be read off its wording; it depends on whether the criterion separates better from worse responses and whether judges score it consistently. Asking the generator for semantically valid, discriminative, judge-stable criteria moved both operational properties only marginally, lifting pairwise accuracy from 48.07% to 48.44%. On 750 preference instances from RubricBench, criteria selected by response calibration alone reached the highest discriminative power and lowest cross-judge variance, but 17.84% of them were semantically invalid; applying semantic filtering before calibration removed the invalid selections and raised average pairwise accuracy to 49.93%.

Once criterion content and criterion-level judgments are fixed, changing the weighting or the hierarchical aggregation barely moves the final evaluation. With identical rubric leaves and identical criterion-level judgments, neither hierarchical aggregation nor global weighting produced consistent gains. With DeepSeek-V4-Flash-GA as judge, flat aggregation reached the highest accuracy at 73.20%, and weighting added only a marginal 0.27 percentage points under Llama3.1-405B-Instruct.

The pipeline targets pairwise preference evaluation of open-ended tasks, the mechanistic analysis rests on 1,500 instances stratified from HealthBench and RubricBench, and the main experiments cover six benchmarks: JudgeBench, RM-Bench Chat, the Chat and Chat-Hard subsets of RewardBench, the Precise-IF and Focus subsets of RewardBench 2, and RubricBench. The extra computation for rubric construction falls mainly in the offline stage, so the authors test distilling the gains into a standalone generator.

In the ablation, removing either semantic validation or response calibration does not lower every benchmark, and without calibration the Focus result matches the full method. Calibration responses are used only for candidate selection and never enter the final evaluation. The authors report that PreciseIF is not the top accuracy setting and note that reward effectiveness depends on more than how often a reward separates responses.