On 23 September 2026 OpenAI published a customer story about Airbnb widening engineering access to GPT-6 Astra, through both the OpenAI API and Amazon Bedrock. The headline figure is an 80% year-on-year increase in features shipped, attributed to Airbnb's CTO. There is also an engineer's anecdote: tasks outside coding that used to take 20 or more rounds of back and forth now take three or four passes.
If you run an engineering team in Sydney or Melbourne and your board has seen that number, you will probably be asked whether you are on the wrong tool. This post is our answer as a Claude-first consultancy, and it applies just as much to the case studies Anthropic publishes about Claude Code.
Should Australian teams switch from Claude Code to GPT-6 Astra based on Airbnb's results?
Not on the strength of that case study alone. The 80% figure is a self-reported number from a customer, published by the vendor, with no stated baseline, method or control group. It tells you Airbnb is happy with GPT-6 Astra. It does not tell you whether your team, on your codebase, would ship more with Astra than with Claude Code. The only reliable way to answer that is a structured trial on your own repositories with metrics you define before you start.
None of that means Airbnb is wrong. A large engineering organisation choosing to widen access is a real signal, and the Bedrock route matters for teams with AWS commitments. It is simply a different kind of evidence from a measurement.
What a features-shipped number can and cannot show
"Features shipped" is an output count. It moves for many reasons that have nothing to do with which model sits in the editor. Before treating any velocity claim as a comparison, ask what else changed in the same year.
Did the definition of a feature change, or did teams start splitting work into smaller releases?
Did headcount, team structure or release cadence change over the same period?
Is the comparison year-on-year against a year with no AI tooling, or against a previous AI tool?
Were defect rates, rollbacks and incident counts tracked alongside the output number?
Was the figure measured across all engineers or drawn from early adopters?
A team moving from no assistant to any capable assistant will often see a large jump. That is a story about adopting AI coding tools, not about one model beating another. We made the same point about vendor-run benchmark tables in our read of OpenAI's GPT-6 Sol benchmarks and in Claude Fable 5.1 vs GPT-6 Astra. A customer testimonial carries even less method than a benchmark table, because nobody outside the company can rerun it.
Reading the evidence by type
Not all vendor material is equal. Here is how we weigh the common kinds when a client asks whether to switch coding tools.
| Evidence type | Who controls it | Can you reproduce it? | Weight in a decision |
|---|---|---|---|
| Customer case study with a % uplift | Vendor and customer | No | Low: a signal of satisfaction |
| Single-engineer anecdote | Vendor marketing | No | Very low |
| Vendor-run benchmark table | Vendor | Partly, if methods are published | Low to medium |
| Independent public benchmark | Third party | Yes | Medium |
| Your own trial on your own repos | You | Yes | High |
The same table applies to Anthropic. When we show clients Claude Code results, we point them to their own pilot data first and the published material second.
How to run a fair Claude Code vs GPT-6 Astra comparison
A useful trial is small, time-boxed and agreed in writing before anyone touches a tool. For a mid-market team this is usually four to six weeks.
Pick the work before the tools
Choose 20 to 40 real tickets from your backlog across a spread of difficulty: bug fixes, a small feature, a refactor, a test-coverage task and one piece of non-coding work such as writing a design note. Split them so each tool gets a comparable mix.
Agree the metrics up front
Track time to merged pull request, review rounds per pull request, defects found in review and in the following two weeks, and engineer-reported effort. The Airbnb anecdote about passes versus rounds is a reasonable thing to measure, so measure it properly: count iterations per task for both tools.
Control what you can
Use the same engineers on both tools where possible, swap halfway, and keep model settings and repository access equivalent. If one tool gets your best engineer and the other gets the newest hire, the result tells you about people, not models.
Price it in Australian dollars
Put the run cost next to the outcome. As an illustrative example only: a ten-engineer team spending $4,500 a month on model usage that saves each engineer three hours a week, at a loaded cost of $120 an hour, recovers roughly $15,600 a month. If the second tool costs $1,500 more and saves one more hour per engineer, that is about $5,200 extra value for the extra spend. Our ROI calculator runs the same sums with your numbers.
Governance questions that matter more than the headline
For regulated Australian organisations, model choice is rarely decided on velocity alone. APRA CPS 234 and CPS 230 expectations, Privacy Act obligations and your own data residency position will shape which hosting route you can use. Both Claude and GPT-6 Astra are available through Amazon Bedrock, so the cloud procurement path may be similar. The differences show up in how each tool handles repository access, logging, permissions and human review of agent-written code.
That review step is where productivity gains are usually won or lost. If agent output piles up waiting for approval, the model does not matter much. We covered this in fixing the review process before a Claude Code modernisation.
What not to conclude
Do not conclude that Airbnb's result proves GPT-6 Astra is better than Claude Code, and do not conclude the opposite either. Do not treat a large uplift in another company as a forecast for yours. And do not let a vendor case study, from OpenAI or Anthropic, replace a trial you could run in a month. A switch costs retraining, new integrations and lost habits, so the bar for evidence should be higher than a single published percentage.
If you want a second pair of hands, we run structured Claude Code evaluations against alternatives on a client's own codebase, with the metrics agreed before the pilot starts. See our services or book a conversation to talk it through.
