If you are going to ask a board to fund Claude Code across an engineering team, you will eventually be asked for evidence. The honest problem is that most of the eye-catching productivity numbers floating around are unmeasured anecdotes: a developer felt faster on a greenfield script and rounded it up to a multiplier. That is not going to survive a finance review. This post is about measuring the real gain in a way you can defend, including the parts where the gain is small or zero.
Why most AI productivity claims fail scrutiny
Three problems sink the typical claim before it reaches the boardroom.
Self-reported time savings inflate under novelty. People feel faster with a new tool, and that feeling fades while the survey numbers do not.
Cherry-picked tasks mislead. A greenfield script is the friendliest possible case and looks nothing like the maintenance-heavy reality most teams live in.
Output volume is not outcome. More pull requests can simply mean more review load, not more value shipped.
The common thread is that all three measure the wrong thing, then report it confidently. If you want a number that holds up, you have to decide what success looks like before you start, and measure the same work the same way on both sides of the change.
Metrics that hold up
A small set of metrics survives scrutiny because they tie to outcomes rather than effort.
Cycle time. Ticket-start to merged, measured before and after on the same team and the same kind of work.
Review turnaround and rework rate. Does AI-assisted code bounce back more often or less? Rework is where hidden costs hide.
Escaped defects. Quality regression is the failure mode to watch. A velocity gain that ships more bugs is not a gain.
Throughput on a fixed scope type. Pick a category like migration tickets or test-coverage tasks and count completions on a stable definition.
One practical note on instrumentation: most of these metrics are already sitting in your version control and ticketing systems, so you rarely need new tooling to capture them. Cycle time, review turnaround, and rework are all derivable from data you already hold. The work is in defining the categories cleanly and resisting the urge to add new dashboards. A spreadsheet and a clear definition will tell you more than an expensive analytics product pointed at fuzzy inputs.
How to run a fair four-week baseline
The method matters more than the tooling. A fair comparison is not complicated, but it does require discipline to leave the variables alone.
Two weeks of instrumented baseline before rollout, with no behaviour change, so you know where you actually started.
Two weeks post-rollout on the same work mix, not a hand-picked friendlier one.
Hold review standards constant across both. Measure the difference, do not vibe it.
If you change the work mix or relax review standards halfway through, you have measured nothing, and a sharp board member will spot it. The whole value of a baseline is that it is boring and consistent.
What realistic numbers look like
Here are the ranges we actually see, stated honestly. On work that suits it, like migrations, boilerplate, and test coverage, cycle-time improvements of 15 to 40 per cent are realistic. On judgement-heavy design work, the gain is close to zero, and pretending otherwise just sets up a disappointment. The blended figure is what matters for the business case.
Put it in money. For a ten-engineer Sydney team with around $2.2M in annual payroll, a 20 per cent improvement on half the work mix is roughly $220,000 a year of recovered capacity. That funds a rollout many times over, and it does so without a single inflated claim. A defensible $220,000 beats an indefensible million every time you are sitting across from finance.
When the numbers say stop
A measurement culture has to be willing to act on bad readings, not just good ones. Pull back, or slow down, when you see defect rates climbing, when review becomes a bottleneck because more code is arriving than the team can properly check, or when junior engineers are leaning on the tool instead of building judgement. Each of those is a signal that the gain is borrowed against future quality. The same metrics that justify the spend will tell you when to ease off, which is exactly why you want them in place from day one.
Every Claude Code rollout we run includes a measurement baseline, so the board sees evidence rather than enthusiasm. We work with engineering teams across Sydney and beyond. If you need a number you can defend, book a call and we will build the baseline in from the start.
