# Conversational Recommendation

This work presents the first user study assessing the reliability of LLM-as-a-judge for evaluating conversational recommendation system (CRS) responses: 20 multi-turn sessions were sampled from TalkPlayData-Challenge, candidate responses were generated by four instruction-tuned LLMs, and 20 music-domain experts produced n=400 ratings on Personalization Quality and Explanation Quality; bootstrapped correlation analysis over 10,000 iterations found moderate positive alignment between LLM judges and human assessments (highest r=0.55 for Gemini-3.1Pro on personalization), outperforming all reference-based baselines.

This work presents the first user study assessing the reliability of LLM-as-a-judge for evaluating conversational recommendation system (CRS) responses: 20 multi-turn sessions were sampled from TalkPlayData-Challenge, candidate responses were generated by four instruction-tuned LLMs, and 20 music-domain experts produced n=400 ratings on Personalization Quality and Explanation Quality; bootstrapped correlation analysis over 10,000 iterations found moderate positive alignment between LLM judges and human assessments (highest r=0.55 for Gemini-3.1Pro on personalization), outperforming all reference-based baselines.