OpenAI Publishes Results From Handing Contract Work to AI: Astra Scores 55.0%
Across 11 research tasks, GPT-6 Astra averaged 55.0%, or 13.4 percentage points above GPT-5.6 Sol. The agents were asked to configure contracts and approval workflows, and OpenAI says complex work still needs human oversight.
On October 6, OpenAI announced a research collaboration with Ironclad, a company that builds contract management software. The two companies trained and evaluated AI agents, systems that carry out work on their own, on tasks involving contract setup, approval workflow configuration, and reusable legal clauses. GPT-6 Astra is the first frontier model OpenAI trained on this work.
How much of the contract workflow was handed over
The research covered the multiple steps that make up contract work. That included setting up non-disclosure agreements, configuring purchase approval workflows, and revising legal clauses to fit a chosen jurisdiction. In other words, the agents were asked to handle not just the drafting of contract language but also the configuration and workflow logic of the software that processes those contracts.
The evaluation covered 11 research tasks. Each task carried between 8 and 50 grading criteria used to assess the results. OpenAI estimated that even an experienced user would need an average of 30 to 40 minutes to finish a single task.
Astra at 55.0%: how much better than Sol?
GPT-6 Astra’s average score was 55.0%. GPT-5.6 Sol scored 41.6%, a gap of 13.4 percentage points. One caveat on the comparison: Astra was run with the Max setting while Sol used the High setting, so the two models generated their answers under different conditions.
Astra was also faster per attempt, averaging 19.2 minutes versus 37.0 minutes for Sol. Those figures are simulated values calculated from assumptions about each model’s processing and generation speed.
On individual tasks, scores came in above the averages. Astra met roughly 94% of the grading criteria in an estimated 20 minutes, while Sol met about 85% in an estimated 32 minutes. An internal model used during Astra’s development scored 63.7% on the same 11 tasks.
So can it be put to work right away?
OpenAI said complex contract work still requires human oversight. The company also disclosed what data was used for training and evaluation. It said it did not use OpenAI customer data or OpenAI’s own internal contracts, and did not use private Ironclad customer data or contracts either.
OpenAI said it is recruiting a small number of software companies to take part in similar research. What makes this collaboration notable is that the AI was tested not only on drafting contract documents but also on configuring approval workflows and setting up contract processes in the software. The questions to watch are whether performance on research tasks carries over to real work, and how much the scope of human review and the time the work takes will change.