Australian software buyers are starting to see "ARC-AGI-2" in vendor pitches and comparison threads without much explanation. GLM-5.2 recently posted the highest ARC-AGI-2 score of any open-weight model, and that number is already showing up in sales decks as proof of superior reasoning. Here's what the test actually checks, and what it leaves out, before it changes a single procurement decision.
What ARC-AGI-2 actually measures
Most AI benchmarks test knowledge or coding. ARC-AGI-2 tests something different: abstract reasoning on puzzles the model has never seen, designed so memorising the training data does not help. In 2026 GLM-5.2 posted the highest ARC-AGI-2 score of any open-weight model, around 22.8%, which sounds low until you know that most models score in the single digits.
For an Australian business owner, the benchmark is easy to misread. A high score does not mean the model is close to human general intelligence. It means the model is better at a specific kind of novel-pattern puzzle.
What the score does tell you:
The model can handle some genuinely new problems, not only ones like its training data.
It has stronger reasoning than raw knowledge tests reveal.
The open-weight field is improving on hard reasoning, not just on coding.
Why the test resists memorising
Older benchmarks reward a model for having seen something similar during training. ARC-AGI-2 is built the other way around: every puzzle combines shapes, colours and rules in a fresh arrangement, so the model has to work out the pattern on the spot instead of recalling a similar example. That is why scores across the board stay low. A model that reaches 20 to 25% is solving problems it has never seen a version of before, which is a different skill to scoring well on a coding or trivia test.
Why a puzzle score is not a buying signal
The gap between a benchmark and your business is wide. ARC-AGI-2 uses abstract grids, not invoices, rosters, or customer emails. A model that scores well on puzzles may still be middling at the plain, repetitive reasoning a Melbourne office actually needs.
What ARC-AGI-2 does not tell you:
How the model handles your documents, tone, and terminology.
How much it costs to run at your volume.
How reliable it is across a long, multi-step task.
When we assess a model for a client, the leaderboard is background, not the decision. A model that tops a chart but fails on your real tasks is worth nothing, while a two-week pilot on your own work costs around $3,000 and answers the question properly.
The practical test is always the same:
Run the model on a sample of your own real work.
Measure accuracy and cost on that sample, not on a public chart.
Choose the model that does your job well, not the one that tops a benchmark.
This is not an argument against benchmarks. They are a fair way to compare models at a glance and to spot which labs are pushing reasoning forward. The mistake is treating a single number as a purchase decision. Use the leaderboard to build a shortlist, then let your own tasks pick the winner from it.
How Automata AI actually tests a model for a client
When a Sydney or Melbourne business asks which model to run, we do not start from a leaderboard. We start from their own work: a batch of real invoices, real customer emails, real rosters, whatever the task actually is. The model runs against that batch, we score it for accuracy and cost, and we repeat with two or three other candidates, including Claude. The exercise usually takes about two weeks and costs in the order of $3,000, and it answers a much narrower, much more useful question than any public benchmark: does this model do your job, at your volume, for a price that makes sense.
What we check instead of a rank
Accuracy on your own documents and terminology, not a generic test set.
Cost per task at the volume you actually run, not a headline API price.
Consistency across a long, multi-step job, not a single best-of-five attempt.
How the vendor handles your data, which matters more once you are past a pilot and into production.
How much engineering effort it takes to connect the model to your existing systems.
None of this makes ARC-AGI-2 a bad benchmark. It is a fair way to see which labs are pushing reasoning forward, and a high score is a reasonable filter for building a shortlist. The mistake is stopping at the shortlist and calling it the decision.
ARC-AGI-2 is a useful signal that open reasoning is improving. It is not a reason to pick a model for your business. To test a model against your actual work rather than a puzzle set, book a session and we will benchmark it on the tasks that pay your bills.
