Blog

Claude vs GPT-6 Astra on Contracts: Test Before You Trust

November 2026 · Technical

Hand-drawn contract page under a magnifying glass with a tick checklist beside it, for testing Claude on contracts
← Back to all posts

On 6 October 2026 OpenAI published a research write-up with Ironclad, the contract management platform, showing its newest agent model working through real contracting tasks. The headline is that GPT-6 Astra beat the older GPT-5.6 Sol by a wide margin and finished faster. For an Australian legal, procurement or finance team looking at agents for contract work, the more useful question is what those numbers can and cannot tell you about your own documents.

We are a Claude consultancy, so read this with that in mind. Our view is not that the OpenAI result is wrong. It is that a vendor benchmark on someone else's contracts is a starting hypothesis, and the work of turning it into a buying decision is the same whichever model you pick.

What did OpenAI actually report about GPT-6 Astra and Ironclad?

OpenAI reports that GPT-6 Astra scored a mean rubric score of 55.0% across 11 real contracting tasks drawn from SEC EDGAR filings, against 41.6% for GPT-5.6 Sol. That is about 32% higher on average. Astra also took 19.2 minutes per attempt compared with 37.0 minutes for Sol, and an unreleased internal OpenAI model reached 63.7%. These are self-reported internal figures that have not been independently checked.

Each task covered legal, commercial or procurement work that would take an experienced person roughly 30 to 40 minutes. Scoring used rubrics with between 8 and 50 criteria per task, depending on complexity. The figures below are as published by OpenAI in its Ironclad research post.

OpenAI's self-reported Ironclad contracting results, October 2026
ModelMean rubric scoreMinutes per attempt
GPT-5.6 Sol41.6%37.0
GPT-6 Astra55.0%19.2
Internal OpenAI dev model63.7%Not stated

Two things stand out. First, the best published model still misses close to half the rubric points on tasks a contracts manager would consider routine. Second, the speed gain is large. Halving the time per attempt changes the economics of review more than a few points of accuracy do, provided the output can be checked quickly.

A 55% score on someone else's contracts

A mean rubric score is an average across criteria, not a pass rate. An agent can score 55% by getting every clause summary right and missing every indemnity cap, or the reverse. For contract work those two failures carry very different risk. Before treating any number like this as a signal, ask for the distribution behind it.

The source documents also matter. SEC EDGAR contracts are US public-company agreements filed with a regulator. They tend to be long, heavily negotiated and drafted under US law. A typical Australian mid-market business deals with supplier terms, leases, franchise agreements and SaaS contracts governed by state law, often with Australian Consumer Law and Privacy Act clauses that do not appear in EDGAR filings at all.

None of this is a criticism of the research design. It is simply the gap every published agent benchmark has: it measures the vendor's chosen tasks, not yours.

Questions to ask before trusting any agent benchmark

We use the same checklist whether the benchmark comes from OpenAI, Google or Anthropic. It applies equally to Claude Code and Cowork results.

  • Whose documents were used, and how close are they to the contracts your team actually handles?

  • Is the score a pass rate or a rubric average, and what does the spread across tasks look like?

  • Which criteria did the model miss most often, and are those the high-consequence ones for you?

  • Was the agent allowed to ask clarifying questions, use tools, or read prior versions of the contract?

  • Can every claim in the agent's output be traced back to a clause, with a page or section reference?

  • Who wrote the rubric, and was the grader a person, a model, or both?

The fifth point does the most work. An agent that is right 60% of the time but cites its sources lets a reviewer confirm or reject each finding in seconds. An agent that is right 70% of the time with no citations forces the reviewer to redo the reading. In practice the first is often the more useful tool.

How Claude fits document-heavy agent work

Claude's strength in this kind of work comes less from a single benchmark and more from how it behaves over a long task. In Claude Code and Cowork, Claude works through a set of files step by step, keeps a running record of what it read and changed, and can be told to quote the clause behind every conclusion. That audit trail is what a legal or procurement lead needs to sign off on agent output.

We compared the earlier OpenAI release in our reading of the GPT-6 Sol benchmarks, and covered the legal product launch in Claude vs OpenAI's Astra for Law. The pattern holds here: the useful comparison is not model against model on a vendor's test, but each model against your own contracts with your own reviewers marking the output.

A worked example: pricing a contract review pilot

Take an illustrative Sydney distributor that reviews around 40 supplier agreements a month. If a contracts manager spends 35 minutes on a first-pass review at a loaded cost of $95 an hour, that is roughly $2,200 a month, or about $26,600 a year, on first-pass reading alone.

A four-week pilot would run 30 of last quarter's agreements through Claude, with the manager scoring each output against a short rubric your team writes: payment terms, liability caps, termination rights, auto-renewal, data handling under the Privacy Act. The pilot answers three questions. Which clauses does the agent get right reliably? Which does it miss? And how long does a human take to check a cited summary compared with reading the contract cold?

If checking takes 10 minutes instead of 35, the saving is real even when the agent misses a third of the criteria, because the reviewer is still the one deciding. If checking takes 30 minutes, the agent is not ready for that document type yet. Either answer is worth more than any published average.

What not to conclude from this research

Do not conclude that GPT-6 Astra is better or worse than Claude at Australian contract work. The research does not include Claude, does not include Australian contracts, and has not been reproduced outside OpenAI. Equally, do not dismiss it. A 32% average gain between two model generations in under a year tells you agent quality on professional documents is moving quickly, and that a pilot run in 2025 may no longer reflect what is possible.

The safe reading is narrow: agents are now fast enough on long documents that review time, not drafting time, has become the bottleneck. Design your pilot around that.

Where to start

If your team is weighing agents for contracts, compliance or procurement, start with a small set of your own documents and a rubric your reviewers trust. Our services cover exactly this kind of scoped pilot, and you can book a short call to talk through which document type to test first.

FAQ

Frequently asked questions

What is GPT-6 Astra?

GPT-6 Astra is an OpenAI agent model that OpenAI reports scored 55.0% on its Ironclad contracting benchmark in October 2026, ahead of GPT-5.6 Sol at 41.6%. The figures are OpenAI's own and have not been independently verified.

What is the Ironclad contracting benchmark?

It is a set of 11 real legal, commercial and procurement tasks built from SEC EDGAR contracts, each taking an experienced person about 30 to 40 minutes. Outputs were scored against rubrics of 8 to 50 criteria per task.

Can AI agents review contracts reliably?

Agents can produce fast first-pass summaries and flag clauses, but published scores show even the best models miss many rubric points. They work best when every finding cites the clause so a person can confirm it quickly.

Is Claude better than GPT-6 Astra for contract review?

No independent comparison exists on the Ironclad tasks, and the research did not test Claude. The reliable way to compare is to run both on a sample of your own contracts and have your reviewers score the output.

Ready to move from AI pilot to production?

We help mid-market Australian businesses deploy AI automations that actually reach production and deliver measurable ROI.