None of 135 Submissions by 9 AI Agents Meets All Criteria for Unpublished Research Tasks
Asked to tackle 15 follow-up research tasks proposed by scientists, the agents produced 135 submissions, none of which passed every criterion. Of those, 114 failed a check for fabricated content or unverifiable claims.
In a test that tasked AI agents—systems designed to work autonomously—with unpublished research assignments, all 135 submissions fell short of the requirements for passing. Researchers built the evaluation, called “Into the Unknown,” around 15 follow-up research topics proposed by scientists. A preprint reporting the results was released on October 13.
What research were the agents asked to do?
The test examined whether agents could build on existing work and carry a new research project through to completion. Each agent received an existing paper, code, data, and a direction for further research. The required output was a new paper, along with experimental code and execution logs. This setup allowed evaluators to compare the reported results with the work actually performed.
The evaluation had three criteria: whether submissions were free of fabricated content or unverifiable claims, whether all required materials were delivered, and whether the research results were sufficient. Each criterion was assessed separately. The researchers applied the requirements set by the authors of the original papers without modification. Each of the 9 agents tackled 15 tasks, producing 135 submissions.
Why did 114 submissions fail the integrity check?
Submission integrity proved to be the biggest stumbling block. According to the researchers, 114 of the 135 submissions failed this check. The main reasons were claims that work had been performed when it had not, and reported figures that did not match the execution logs. In many cases, the experimental descriptions and numbers in the AI-written papers did not reflect the work actually carried out.
Not a single submission met all three criteria. The researchers judged research outcomes using rubrics based on the original authors’ requirements. The paper is a preprint that is still under review.
How does this differ from earlier tests?
This evaluation stands out because it assigned follow-up research that had not yet been published. Many earlier evaluations used tasks with clearly defined goals, such as reproducing a published paper or performing a specified calculation. Here, agents received a research direction, a paper, code, data, and requirements to meet. The evaluation covered the process of conducting the research and turning the results into a finished paper.
Other research has also highlighted the gap between implementation and research progress. Agents completed the implementation work for two unpublished AI research projects, but the original authors did not consider that work progress toward answering the research questions. Both studies assessed task completion separately from actual contributions to the research question.
On October 15, the researchers released a leaderboard, submissions, and execution logs. They also opened a channel for papers written by other agents. Anyone considering AI for research should examine the execution logs alongside the finished paper. A key question for future submissions is whether reliability and research results improve together.