A novelty judge can look reliable until the prompt changes: GPT-5.4 falls from 93.5% to 40.9% on the same idea pairs
Synopsis
Ai2 released Old Ideas, Novel Problems, a study, benchmark and data pipeline testing whether large language models can judge the novelty of research ideas. Across six judges, changing one setting such as the prompt shifted verdicts sharply: GPT-5.4 fell from 93.5% to 40.9% accuracy on human-versus-generated idea pairs, below chance, and purpose-built novelty evaluators were neither more accurate nor cheaper than a simple prompted Claude Opus 4.6 baseline.
Interpretation
Judging whether a research idea is novel is increasingly handed to another large language model, called a novelty judge, and Ai2's tests show the same ideas receive very different verdicts under different evaluation setups. Such judges were often validated only on human-written papers, while they are increasingly asked to evaluate AI-generated ideas. Across six judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6), changing a single factor such as the prompt produced large shifts in judgment.
Prompting produced the largest shifts: the same pair of ideas can move a model from near-perfect to below-chance accuracy. On the same Human+Generated pairs, GPT-5.4 reached 93.5% accuracy when told exactly one idea in each pair was judged novel and asked which, and dropped to 40.9% when simply asked which idea was novel, changing its verdict on more than half the pairs. On the Human-Only set of human-authored ideas with stronger and weaker novelty evidence, the same prompt change moved accuracy by only about seven points, yet Claude Opus 4.6 still shifted 21 points there.
More retrieval and reasoning did not necessarily improve novelty judgments, and purpose-built evaluators were not more accurate. On Human+Generated, a simple Claude Opus 4.6 prompt with low reasoning and majority voting over three calls reached 0.854 macro-F1 at $2.59 total cost; Idea Novelty Checker scored 0.642 at $30.66 and Sakana AI's AI Scientist 0.772 at $82.05. A GPT-5.6 Sol agentic judge with extra-high reasoning and live iterative arXiv search scored 0.78 macro-F1, behind Claude Opus 4.6 at 0.89, at roughly 10x the cost per decision.
Perspective
The benchmark targets researchers and system developers who need to evaluate the novelty of AI-generated research ideas; it tests six general-purpose LLM judges, two purpose-built novelty evaluators and one retrieval-equipped agentic judge, using human idea pools mined from ICLR 2026 reviews and ideas from a simple LLM generator.
The assumption that generated ideas are less novel than their human-authored counterparts is a weak label rather than a verified judgment for every idea; the authors note many core findings do not depend on labels, since two reasonable setups giving opposite answers for the same pair already shows instability. LLM-based novelty evaluation remains an open problem.