LLM novelty judges flip on a few prompt words: GPT-5.4 falls from 93.5% to 40.9% accuracy
Synopsis
Ai2 released Old Ideas, Novel Problems, a study with a benchmark, data, code and an automated pipeline for testing whether large language models can reliably judge the novelty of research ideas. The study mines ICLR 2026 peer reviews for 154 higher-novelty and 145 lower-novelty human ideas, and pairs 154 human ideas with 154 ideas from a simple generator. Across six judge models, changing only prompt wording dropped GPT-5.4 accuracy on the human-plus-generated pairs from 93.5% to 40.9%, and retrieval or extra reasoning did not consistently improve judgments.
Interpretation
Judging whether a research idea is new is handed to another large language model, called a novelty judge, which assesses one idea alone or compares two ideas against prior work. There was no standard way to build a novelty judge; teams choose a model, prompt, evaluation format, reasoning budget and sometimes a literature-retrieval system, and those choices can materially affect the verdict. These judges are often validated only on human-written papers rather than the AI-generated ideas they are actually meant to evaluate.
The same ideas receive very different judgments under seemingly small changes to the evaluation setup. On the human-plus-generated pairs, GPT-5.4 saw the same pairs under two prompts: one said reviewers found exactly one idea in each pair novel and asked which, the other simply asked which idea is novel; accuracy fell from 93.5% to 40.9%, below chance, and the model changed its verdict on more than half the pairs. The same prompt change shifted accuracy by only about seven percentage points on the human-only dataset, which also showed substantial drift, with the same change moving Claude Opus 4.6 accuracy by 21 points.
More search and more reasoning do not necessarily produce better novelty judgments. GPT-5.6 Sol, given an extra-high reasoning budget and live iterative search restricted to arXiv, scored 0.78 macro-F1 on the human-plus-generated set, 11 points behind Claude Opus 4.6 at 0.89 and in roughly the same range as simpler Claude Opus 4.5 and GPT-5.4 setups, while costing roughly 10 times more per decision at current API prices. All judges in this comparison ran once per idea, and macro-F1 measures how well a judge identifies novel and non-novel ideas, higher being better.
Purpose-built novelty evaluators were not more accurate, despite costing substantially more. On the human-plus-generated set, a simple Claude Opus 4.6 prompt with low reasoning and majority voting across three calls achieved 0.854 macro-F1 at a total evaluation cost of $2.59; the literature-grounded Idea Novelty Checker scored 0.642 at $30.66, and Sakana AI's AI Scientist, which iteratively searches Semantic Scholar, scored 0.772 at $82.05. Both dedicated evaluators used Claude Opus 4.6 with one call per decision.
Perspective
The evaluation covers 154 higher-novelty and 145 lower-novelty human ideas mined from ICLR 2026 peer reviews, plus 154 human ideas paired with 154 ideas from a simple generator; the generated ideas are assumed less novel than their human counterparts, which are weak labels.
In the human-plus-generated setting the novelty of generated ideas is assumed rather than verified idea by idea; the expert review of 30 pairs was weighted toward cases the strongest judge got wrong; and the findings point to novelty evaluation remaining an open problem, with future work needing to test judges on the kinds of ideas they will actually evaluate and to track prompt and model sensitivity.