# LLM-based novelty evaluation of research ideas

Ai2 released Old Ideas, Novel Problems, a study, benchmark and data pipeline testing whether large language models can judge the novelty of research ideas. Across six judges, changing one setting such as the prompt shifted verdicts sharply: GPT-5.4 fell from 93.5% to 40.9% accuracy on human-versus-generated idea pairs, below chance, and purpose-built novelty evaluators were neither more accurate nor cheaper than a simple prompted Claude Opus 4.6 baseline.

Ai2 released Old Ideas, Novel Problems, a study, benchmark and data pipeline testing whether large language models can judge the novelty of research ideas. Across six judges, changing one setting such as the prompt shifted verdicts sharply: GPT-5.4 fell from 93.5% to 40.9% accuracy on human-versus-generated idea pairs, below chance, and purpose-built novelty evaluators were neither more accurate nor cheaper than a simple prompted Claude Opus 4.6 baseline.