Skip to main content

# Old Ideas, Novel Problems

2 related articles

Subscribe to updates

Ai2 Blog Oct 27, 2026
A novelty judge can look reliable until the prompt changes: GPT-5.4 falls from 93.5% to 40.9% on the same idea pairs
Ai2 released Old Ideas, Novel Problems, a study, benchmark and data pipeline testing whether large language models can judge the novelty of research ideas. Across six judges, changing one setting such as the prompt shifted verdicts sharply: GPT-5.4 fell from 93.5% to 40.9% accuracy on human-versus-generated idea pairs, below chance, and purpose-built novelty evaluators were neither more accurate nor cheaper than a simple prompted Claude Opus 4.6 baseline.
Ai2 released Old Ideas, Novel Problems, a study, benchmark and data pipeline testing whether large language models can judge the novelty of research ideas. Across six judges, changing one setting such as the prompt shifted verdicts sharply: GPT-5.4 fell from 93.5% to 40.9% accuracy on human-versus-generated idea pairs, below chance, and purpose-built novelty evaluators were neither more accurate nor cheaper than a simple prompted Claude Opus 4.6 baseline.
Old Ideas, Novel Problems LLM-based novelty evaluation of research ideas Large language models
Hugging Face Blog Oct 28, 2026
LLM novelty judges flip on a few prompt words: GPT-5.4 falls from 93.5% to 40.9% accuracy
Ai2 released Old Ideas, Novel Problems, a study with a benchmark, data, code and an automated pipeline for testing whether large language models can reliably judge the novelty of research ideas. The study mines ICLR 2026 peer reviews for 154 higher-novelty and 145 lower-novelty human ideas, and pairs 154 human ideas with 154 ideas from a simple generator. Across six judge models, changing only prompt wording dropped GPT-5.4 accuracy on the human-plus-generated pairs from 93.5% to 40.9%, and retrieval or extra reasoning did not consistently improve judgments.
Ai2 released Old Ideas, Novel Problems, a study with a benchmark, data, code and an automated pipeline for testing whether large language models can reliably judge the novelty of research ideas. The study mines ICLR 2026 peer reviews for 154 higher-novelty and 145 lower-novelty human ideas, and pairs 154 human ideas with 154 ideas from a simple generator. Across six judge models, changing only prompt wording dropped GPT-5.4 accuracy on the human-plus-generated pairs from 93.5% to 40.9%, and retrieval or extra reasoning did not consistently improve judgments.
Old Ideas, Novel Problems LLM-based novelty judgment of research ideas Large language models

You’re all caught up