There is a real gap between a model that tops a benchmark and a model you would leave running unattended on production systems overnight. Anthropic's recent case study with Cognition, the company behind the autonomous software engineer Devin, is a clear picture of that gap. For Australian businesses weighing when to let an agent off the leash, it is one of the more useful signals to come out of the last month.
The people who are hard to impress
Silas Alberti, SVP of Research at Cognition, has tested nearly every Claude model inside Devin. By his account, Claude Fable 5 is the first he would trust to leave running overnight. That verdict carries weight precisely because his team is hard to please. They have watched models score well on a benchmark and then fall apart the moment engineers pointed them at real work. As Alberti puts it, they have been burned like that a number of times.
Devin takes on the work engineers never quite get to: codebase migrations, the backlog of small bugs, the features that keep slipping. The code it writes has to be reliable and production ready, for customers that range from fast-growing startups to large enterprises. Cognition traces the first real jump to an earlier Claude Sonnet release that could reliably chain tools and hold a multi-step task together. When they plugged it into Devin, internal usage tripled.
That tripling is the tell. Adoption inside a demanding engineering team did not follow a benchmark headline. It followed the moment the model stopped needing a babysitter. Capability got the team interested; reliability got them to hand over real work.
Why a benchmark score is the wrong question
A benchmark tells you a model can do something once, under test conditions, with someone watching. Reliability tells you it will do the same thing the same way at 3am on run number four hundred, with nobody watching. Those are different properties, and only the second one lets you actually walk away. An overnight agent that is brilliant 95 percent of the time and quietly wrong the other 5 percent is not a time-saver. It is a liability you have to audit every morning.
For a business, the cost of that unreliability is concrete. Say a Sydney firm points an agent at overnight invoice reconciliation to save a bookkeeper two hours a day. If one run in twenty mis-posts entries and nobody catches it for a week, the clean-up can wipe out a month of saved time and put a $45,000 error in front of the accountant. The value was never the raw speed. It was being able to trust the output without re-checking every line.
What reliability actually looks like
Holding a long task together: the agent keeps context across many steps instead of losing the thread halfway through.
No quiet failures: when it cannot do something safely, it stops and flags a person rather than guessing and moving on.
The same behaviour every run: the output at 3am on a Tuesday matches the output you approved in testing.
Evidence you can glance at: each run leaves behind logs or a short recording, so approval takes seconds instead of a line-by-line review.
How Australian teams should gate an overnight agent
The practical takeaway is to treat autonomy as something an agent earns per task, not a switch you flip once. Start with a person reviewing every output. Tighten the guardrails around what the agent may and may not do on its own. Watch how it behaves on real work for a few weeks. Only then extend the leash to the specific tasks where it has proven it will not surprise you.
That staged approach matters more in regulated settings. If your overnight process touches customer records, an agent running unattended sits squarely inside your Privacy Act obligations, so the guardrails are not optional polish. A reasonable first project is a low-stakes maintenance loop with a clear finish line, where a bad run costs an hour rather than a client. Prove the pattern there, measure it, and widen scope only once the numbers hold. A team that does this well can move from supervised runs to trusted overnight work inside a quarter, often for less than the $120K a single senior hire would cost.
The headline is not that one model is the smartest. It is that reliability, not a leaderboard position, is what unlocks unattended work. Automata AI helps Australian businesses decide which tasks are safe to hand to an autonomous Claude agent, and how to gate the ones that are not. Book a brainstorm to map your first overnight workflow.
