# PlaylistBench

The work introduces PlaylistEval, an agentic framework that builds video-language judge benchmarks without human annotation by automatically generating questions whose evidence is scattered across two distant segments of ~100-hour playlists and whose wrong answers differ only visually, yielding PlaylistBench with 630 pairs across seven domains; on a stratified subset of 152 pairs it agrees with human judgments 93.0% of the time (IAA 0.781), while evaluating 17 omnimodal and multimodal models from eight families shows frontier judges reach only 75.4% pairwise accuracy, open-source judge models perform far behind, retrieval improves accuracy by up to 10.5 points yet the best retriever finds both relevant segments in the top 10 only 37.

The work introduces PlaylistEval, an agentic framework that builds video-language judge benchmarks without human annotation by automatically generating questions whose evidence is scattered across two distant segments of ~100-hour playlists and whose wrong answers differ only visually, yielding PlaylistBench with 630 pairs across seven domains; on a stratified subset of 152 pairs it agrees with human judgments 93.0% of the time (IAA 0.781), while evaluating 17 omnimodal and multimodal models from eight families shows frontier judges reach only 75.4% pairwise accuracy, open-source judge models perform far behind, retrieval improves accuracy by up to 10.5 points yet the best retriever finds both relevant segments in the top 10 only 37.