Rewording the Prompt Cuts AI Novelty-Judging Accuracy From 93% to 41%
Ai2 put AI novelty reviewers to the test. Judging the same ideas, GPT-5.4's accuracy slid from 93.5% to 40.9% depending on how the prompt was worded.
Ai2 released a study called “Old Ideas, Novel Problems” on October 27 (local time). It examines how consistently AI systems can judge whether a research idea is actually original. Alongside the paper, the team published benchmark data, code, and a tool that automatically generates new evaluation sets. The researchers report that even small changes to the prompt given to a model produced sharply different verdicts on the very same idea.
Does rewording the prompt change the verdict?
The team tested GPT-5.1, GPT-5.2, and GPT-5.4 along with Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. The models were asked either to assign a novelty score to a single idea or to pick the more novel of two ideas. The researchers then varied one factor at a time, including the wording of the prompt, how much reasoning budget the model was given, and whether related papers were supplied, to see how much the judgments moved.
GPT-5.4 proved especially sensitive to prompt wording when comparing a human-written idea against an AI-generated one. When the prompt stated that “reviewers judged exactly one of the two to be novel,” accuracy was 93.5%. When the model was simply asked which idea was novel, accuracy fell to 40.9%, a gap of 52.6 percentage points. On more than half of the pairs, the model flipped its choice. When two human ideas were compared, the same change in wording shifted results by only about 7 percentage points.
How were the right answers established?
The evaluation labels were drawn from peer reviews submitted to ICLR 2026. For the high-novelty side, the researchers selected 154 ideas from accepted papers that ranked in the top 10% of ratings within their field, specifically cases where a reviewer praised the work’s originality and no other reviewer disputed that assessment. For the other side, they collected 145 ideas from rejected papers in the bottom 10% where reviewers raised doubts about novelty and no reviewer praised its originality.
Image: huggingface.co
For the test involving machine-generated ideas, the 154 selected human ideas were pitted against 154 AI-generated ones. The AI was given only a research field and an instruction to propose a new idea, with no ability to search the literature or revise its output. The researchers assumed these AI-generated ideas were the less novel of each pair and said they did not verify that assumption case by case. In a separate expert review of 30 pairs, the human idea was rated higher every time. That sample, however, was skewed toward cases where the best-performing evaluator model had gotten the answer wrong.
Does a more expensive evaluation mean a more accurate one?
Spending more on search and reasoning did not reliably improve results. The researchers gave GPT-5.6 Sol a generous reasoning budget and the ability to search the preprint site arXiv iteratively. Its overall score for separating novel ideas from unoriginal ones was 0.78. Claude Opus 4.6, which read only five pre-retrieved abstracts, scored higher at 0.89. According to the researchers, based on developer API pricing at the time, the costlier setup cost roughly 10 times as much per judgment.
Image: huggingface.co
In another comparison, Claude Opus 4.6 with nothing more than a simple prompt scored 0.854 by taking a majority vote across three responses, at a total evaluation cost of $2.59. Purpose-built evaluation tools running on the same model fared worse: the Idea Novelty Checker scored 0.642 at $30.66, and Sakana AI’s AI Scientist scored 0.772 at $82.05. The specialized tools were run once per judgment. In this test, the simple approach won on both accuracy and cost.
The practical takeaway is that when you ask an AI to help pick a research direction, the wording of your prompt and the framing of the task largely determine the answer you get. Evaluation behavior that looked stable on human-written papers did not carry over to AI-generated ideas, the researchers explained. They noted that the released tooling can be applied to other academic communities and to fresh batches of peer reviews.