On 6 October 2026 OpenAI published a research write-up with Ironclad, the contract management platform, showing its newest agent model working through real contracting tasks. The headline is that GPT-6 Astra beat the older GPT-5.6 Sol by a wide margin and finished faster. For an Australian legal, procurement or finance team looking at agents for contract work, the more useful question is what those numbers can and cannot tell you about your own documents.
We are a Claude consultancy, so read this with that in mind. Our view is not that the OpenAI result is wrong. It is that a vendor benchmark on someone else's contracts is a starting hypothesis, and the work of turning it into a buying decision is the same whichever model you pick.
What did OpenAI actually report about GPT-6 Astra and Ironclad?
OpenAI reports that GPT-6 Astra scored a mean rubric score of 55.0% across 11 real contracting tasks drawn from SEC EDGAR filings, against 41.6% for GPT-5.6 Sol. That is about 32% higher on average. Astra also took 19.2 minutes per attempt compared with 37.0 minutes for Sol, and an unreleased internal OpenAI model reached 63.7%. These are self-reported internal figures that have not been independently checked.
Each task covered legal, commercial or procurement work that would take an experienced person roughly 30 to 40 minutes. Scoring used rubrics with between 8 and 50 criteria per task, depending on complexity. The figures below are as published by OpenAI in its Ironclad research post.
| Model | Mean rubric score | Minutes per attempt |
|---|---|---|
| GPT-5.6 Sol | 41.6% | 37.0 |
| GPT-6 Astra | 55.0% | 19.2 |
| Internal OpenAI dev model | 63.7% | Not stated |
Two things stand out. First, the best published model still misses close to half the rubric points on tasks a contracts manager would consider routine. Second, the speed gain is large. Halving the time per attempt changes the economics of review more than a few points of accuracy do, provided the output can be checked quickly.
A 55% score on someone else's contracts
A mean rubric score is an average across criteria, not a pass rate. An agent can score 55% by getting every clause summary right and missing every indemnity cap, or the reverse. For contract work those two failures carry very different risk. Before treating any number like this as a signal, ask for the distribution behind it.
The source documents also matter. SEC EDGAR contracts are US public-company agreements filed with a regulator. They tend to be long, heavily negotiated and drafted under US law. A typical Australian mid-market business deals with supplier terms, leases, franchise agreements and SaaS contracts governed by state law, often with Australian Consumer Law and Privacy Act clauses that do not appear in EDGAR filings at all.
None of this is a criticism of the research design. It is simply the gap every published agent benchmark has: it measures the vendor's chosen tasks, not yours.
Questions to ask before trusting any agent benchmark
We use the same checklist whether the benchmark comes from OpenAI, Google or Anthropic. It applies equally to Claude Code and Cowork results.
Whose documents were used, and how close are they to the contracts your team actually handles?
Is the score a pass rate or a rubric average, and what does the spread across tasks look like?
Which criteria did the model miss most often, and are those the high-consequence ones for you?
Was the agent allowed to ask clarifying questions, use tools, or read prior versions of the contract?
Can every claim in the agent's output be traced back to a clause, with a page or section reference?
Who wrote the rubric, and was the grader a person, a model, or both?
The fifth point does the most work. An agent that is right 60% of the time but cites its sources lets a reviewer confirm or reject each finding in seconds. An agent that is right 70% of the time with no citations forces the reviewer to redo the reading. In practice the first is often the more useful tool.
How Claude fits document-heavy agent work
Claude's strength in this kind of work comes less from a single benchmark and more from how it behaves over a long task. In Claude Code and Cowork, Claude works through a set of files step by step, keeps a running record of what it read and changed, and can be told to quote the clause behind every conclusion. That audit trail is what a legal or procurement lead needs to sign off on agent output.
We compared the earlier OpenAI release in our reading of the GPT-6 Sol benchmarks, and covered the legal product launch in Claude vs OpenAI's Astra for Law. The pattern holds here: the useful comparison is not model against model on a vendor's test, but each model against your own contracts with your own reviewers marking the output.
A worked example: pricing a contract review pilot
Take an illustrative Sydney distributor that reviews around 40 supplier agreements a month. If a contracts manager spends 35 minutes on a first-pass review at a loaded cost of $95 an hour, that is roughly $2,200 a month, or about $26,600 a year, on first-pass reading alone.
A four-week pilot would run 30 of last quarter's agreements through Claude, with the manager scoring each output against a short rubric your team writes: payment terms, liability caps, termination rights, auto-renewal, data handling under the Privacy Act. The pilot answers three questions. Which clauses does the agent get right reliably? Which does it miss? And how long does a human take to check a cited summary compared with reading the contract cold?
If checking takes 10 minutes instead of 35, the saving is real even when the agent misses a third of the criteria, because the reviewer is still the one deciding. If checking takes 30 minutes, the agent is not ready for that document type yet. Either answer is worth more than any published average.
What not to conclude from this research
Do not conclude that GPT-6 Astra is better or worse than Claude at Australian contract work. The research does not include Claude, does not include Australian contracts, and has not been reproduced outside OpenAI. Equally, do not dismiss it. A 32% average gain between two model generations in under a year tells you agent quality on professional documents is moving quickly, and that a pilot run in 2025 may no longer reflect what is possible.
The safe reading is narrow: agents are now fast enough on long documents that review time, not drafting time, has become the bottleneck. Design your pilot around that.
Where to start
If your team is weighing agents for contracts, compliance or procurement, start with a small set of your own documents and a rubric your reviewers trust. Our services cover exactly this kind of scoped pilot, and you can book a short call to talk through which document type to test first.



