# LLM-based novelty judgment of research ideas

Ai2 released Old Ideas, Novel Problems, a study with a benchmark, data, code and an automated pipeline for testing whether large language models can reliably judge the novelty of research ideas. The study mines ICLR 2026 peer reviews for 154 higher-novelty and 145 lower-novelty human ideas, and pairs 154 human ideas with 154 ideas from a simple generator. Across six judge models, changing only prompt wording dropped GPT-5.4 accuracy on the human-plus-generated pairs from 93.5% to 40.9%, and retrieval or extra reasoning did not consistently improve judgments.

Ai2 released Old Ideas, Novel Problems, a study with a benchmark, data, code and an automated pipeline for testing whether large language models can reliably judge the novelty of research ideas. The study mines ICLR 2026 peer reviews for 154 higher-novelty and 145 lower-novelty human ideas, and pairs 154 human ideas with 154 ideas from a simple generator. Across six judge models, changing only prompt wording dropped GPT-5.4 accuracy on the human-plus-generated pairs from 93.5% to 40.9%, and retrieval or extra reasoning did not consistently improve judgments.