400 ratings from 20 music-domain experts show LLM judges align only moderately with humans (up to r=0.55) yet far exceed BLEU and ROUGE-L reference baselines

LLM-as-a-judge shows moderate positive correlation with expert human ratings in CRS response evaluation and consistently outperforms reference-based metrics. Whether LLM judges align with human judgment in conversational recommendation was previously an open question; this work provides the first empirical user-study evidence. 20 music-domain experts rated 80 (session, response) pairs, yielding n=400 ratings, analyzed with 10,000 bootstrap iterations and 95% confidence intervals; all LLM judge configurations show reliably positive Pearson r, while the best reference baseline reaches only r=0.19.

At a fixed 4B scale within the Qwen3 family, explicit judging (r=0.45) outperforms reference-based embedding (r=0.19) and reference-free embedding (r=0.16/0.30). This controlled comparison separates reference dependency, contextual conditioning, and explicit reasoning, showing the gain comes from explicit scoring rather than context alone. Matched model family and parameter scale with bootstrap confidence intervals; reference-free embedding did not help personalization, improved explanation from r=0.02 to r=0.30, but explicit judging still led at r=0.45.

Judge scale and conditioning information jointly determine alignment: larger judges align better, conversation history contributes most for personalization, and domain in-context examples contribute most for explanation. A cumulative prompt ablation quantifies the marginal contribution of each conditioning element, yielding actionable deployment guidance. Cumulative ablation with Gemini-3.1Flash-Lite: adding conversation history raised personalization r from 0.28 to 0.41 (Δ=+0.13), while user profile added only Δ=+0.02; for explanation, in-context examples raised r from 0.38 to 0.46 (Δ=+0.08).

A case study shows judges are robust to semantic inversion and prompt injection but sensitive to writing style and redundant length. Correlation captures average alignment only; this diagnostic case reveals systematic bias patterns under specific edits. Four controlled edits around one response rated 3.75/4.25 by humans: semantic inversion and injection were floored to 1/1 by every judge; ornate rephrasing pushed lighter judges above the human rating (Flash-Lite 4/2→5/4), while redundant padding lowered explanation scores by one to two points.

The results apply to two dimensions of response generation quality in conversational music recommendation—Personalization Quality and Explanation Quality—and suit researchers and engineering teams seeking a low-cost LLM-judge proxy for human evaluation. The conditioning design (user profile, full dialogue history, domain rubric, in-context examples) can serve directly as a deployment template; the ablation identifies multi-turn conversation history as the most consequential input for personalization and domain-anchored examples as the primary lever for explanation. The study builds on 20 multi-turn sessions (turns 2 to 8) from TalkPlayData-Challenge and candidate responses from four open-source instruction-tuned models, so its intended setting is synthetic multi-turn music recommendation dialogue with moderate variation in response quality.

Inter-annotator agreement is moderate (α=0.4478 and 0.4484), so the human reference signal itself carries noise, placing an effective ceiling on any automated metric's correlation; even the best judge reaches only a bootstrapped mean r=0.55 (personalization) and r=0.51 (explanation), which the text reads as a signal that high-stakes scenarios still need a human in the loop. The framework covers only two quality dimensions, leaving factual accuracy and conversational naturalness unaddressed. The bias case study rests on controlled edits to a single response, so how broadly those patterns generalize remains an open question. In addition, reference baselines depend on synthetic reference responses generated by Gemini 2.5-Flash, whose quality may not always reflect what human annotators consider ideal.