Why test the judge
Getting AI to propose research ideas is no longer hard; deciding whether those ideas are actually new is. Many automated research systems hand that step to another model, a "novelty judge". Old Ideas, Novel Problems, a study released by the Allen Institute for AI (Ai2) on October 27, points out that there is no standard way to build one: the model, prompt, reasoning budget and whether to add literature retrieval all vary, and most judges have only been validated on human-written papers, not on the AI-generated ideas they are actually meant to evaluate.
Rather than treating paper acceptance as a proxy for novelty, the team mined ICLR 2026 reviews for passages where reviewers explicitly discussed originality. The high-novelty pool is 154 ideas from accepted papers in the top 10% of their area whose reviewers praised novelty with no one disputing it; the low-novelty pool is 145 ideas from rejected papers in the bottom 10% whose reviewers raised originality concerns with no one praising it. A second setting pairs those 154 human ideas with 154 ideas from a simple LLM generator that had no search and no iteration. An expert checked 30 pairs and preferred the human idea in every one.
Results: change the prompt, flip the verdict
The judges tested were GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The prompt mattered most. On the same human-versus-generated pairs, telling GPT-5.4 that "reviewers found exactly one idea in each pair to be novel" gave 93.5% accuracy; simply asking which idea was novel dropped it to 40.9%, below chance, and flipped more than half of the verdicts. The same change moved accuracy only about 7 points on human-only pairs, though it still shifted Claude Opus 4.6 by 21 points there. There is no universally better prompt either: dropping an instruction not to treat incremental contributions as novel helped some judges and hurt others.
More search and reasoning didn't help. GPT-5.6 Sol with an extra-high reasoning budget and live, iterative arXiv search scored 0.78 macro-F1, 11 points behind Claude Opus 4.6 at 0.89, at roughly 10x the cost per decision. Purpose-built evaluators fared no better: a simply prompted Opus 4.6 with low reasoning and a three-call majority vote scored 0.854 at a total cost of $2.59; Idea Novelty Checker, a literature retrieval and reranking pipeline, scored 0.642 for $30.66; and the novelty evaluator in Sakana's AI Scientist scored 0.772 for $82.05.
For teams building automated research
The most direct warning: if you use an LLM judge to compare two idea-generation systems, changing the prompt can flip a quarter to a third of the verdicts, which is exactly the range the study measured. Ai2 recommends testing judges on the kind of ideas they will actually evaluate and reporting sensitivity across prompts and models. The benchmark, data and the pipeline for building new evaluation sets are public; the paper is arXiv:2610.02022.
via: Ai2: Can we trust LLMs to judge whether a research idea is novel?, arXiv: Old Ideas, Novel Problems