A flaky test is worse than a broken one. A broken test tells you something is wrong. A flaky test tells you nothing, and after the third false red build in a week, most Australian engineering teams start clicking re-run without reading the log. That habit is how a genuine regression slips through six weeks later.
Why flaky tests survive in most engineering teams
Flaky tests hang around because fixing them properly is slower and less rewarding than working around them. A re-run button is one click. Finding the actual race condition buried in a test that touches a shared database fixture is an afternoon nobody wants to spend. So the flaky test gets a retry annotation, a comment saying 'known flaky, ignore', or quietly gets skipped in CI. Six months later a Sydney SaaS team we spoke with had 60-plus tests marked skip, and two of them were quietly not testing a billing edge case that shipped a real bug to production.
Timing and race conditions: assertions that fire before an async operation resolves
Shared test state: one test leaves data behind that the next test depends on not existing
Unseeded randomness: date/time, UUID, or random-sample generation without a fixed seed
External calls: a test that hits a real API or database and inherits that service's own flakiness
CI resource contention: tests that pass locally but fail under load on a shared runner
How Claude Code actually hunts a flake
The productive way to use Claude Code here is not to ask it to fix the test once. It is to ask it to reproduce the failure reliably first, because a flake you cannot reproduce is a flake you cannot verify you have fixed. A working pattern: run the suspect test 50 to 100 times in a loop, capture full output on every failure, and hand Claude Code both the passing and failing logs side by side. It is much better at spotting the difference between two log dumps than at guessing from a single failure.
From there it typically walks the same path a senior engineer would: check git blame on the test and the code it exercises for anything that landed in the last few weeks, look for shared fixtures or global state, and check whether the assertion is racing an async call that has no explicit await. Claude Code will usually propose the fix as a diff with a one-line reason attached, which matters because a flaky-test fix that nobody understands just becomes the next flaky test.
Getting the loop running in your own CI
Setting this up doesn't need new tooling. Most test runners already support a repeat-count flag (Jest, pytest, and Go's test runner all do), so the loop itself is a one-line CI job that runs the suspect test 100 times and saves every failing run's output to a file. Point Claude Code at that output directory with instructions to diff the passing and failing runs and propose a root cause before touching any code. Teams that skip straight to 'just fix it' get worse results than teams that force the diagnosis step first, because the diagnosis is what makes the eventual fix explainable to the rest of the team.
What a real hunt looked like, and what this approach is not
A Melbourne fintech engineering lead ran this pattern across one sprint on a test suite with 40 known-flaky tests dragging out CI runs by close to 20 minutes a day. By the end of the sprint 31 were fixed outright and 6 were reclassified as genuine, intermittent third-party API failures worth mocking instead of chasing. The team put the CI time saved at roughly $45,000 a year once you price out engineer-hours lost to waiting on builds and re-running failed pipelines across a team of twelve.
This is not a substitute for fixing bad test architecture. If your suite shares a single test database across parallel workers, Claude Code can help you diagnose that fifty times over and the real fix is still parallel test isolation, not another patch. And it will not help with a test that fails every time under the same conditions, because that is a bug, not a flake, and treating it as one just delays a fix. Where it earns its keep is the tedious, high-volume detective work: running the loop, reading the diff, and proposing a specific, explainable change instead of a broader rewrite.
If your team is carrying a backlog of skipped or quarantined tests, that backlog is itself the risk, because every skip is a gap in your safety net that nobody is actively tracking. Automata AI helps Australian engineering teams wire Claude Code into CI properly, including the flaky-test triage loop above. Get in touch via /contact if a growing skip-list is starting to worry you.
