Running Claude Code as a reviewer on every pull request is a different problem to running it interactively at your desk. Interactively, you can course-correct mid-task. In CI, the agent runs once, unattended, against a diff it has never seen, and either produces something useful or it doesn't. Getting that reliable takes a narrower scope than most teams start with.
What actually works as a CI reviewer
The teams getting real value out of this are not asking Claude Code to approve or block merges. They're using it for a specific, bounded review pass: does this diff introduce an obvious security issue, does it break an established pattern in the codebase, does it touch a file with a known-fragile history. That's a narrower brief than 'review this PR', and narrower briefs are what make an unattended agent reliable rather than noisy.
Scope the prompt to specific check categories, not an open-ended code review
Give it the diff plus surrounding file context, not just the changed lines
Post findings as PR comments, never as an auto-approve or auto-merge action
Track false-positive rate per check category and prune the ones that aren't earning their keep
The failure modes worth planning for
The most common early mistake is giving the CI agent too much latitude and getting a wall of stylistic nitpicks on every PR, which developers learn to ignore within a fortnight. At that point it has actively made review worse, not better. The fix is the same discipline you'd apply to a human reviewer: a small number of things it checks well, checked consistently, rather than an open brief to comment on anything.
A Canberra-based dev team running this on a 15-engineer codebase started with an unscoped prompt and got an average of 9 comments per PR, most of them ignored. After narrowing the brief to three categories (auth and permission changes, database migration safety, and secrets left in a diff), comment volume dropped to roughly 1.5 per PR and the acceptance rate, meaning comments actually acted on, went from under 10% to over 60%. They estimated the migration-safety check alone had likely prevented one production incident that would have cost a weekend of unplanned work, conservatively worth $6,000 in overtime and opportunity cost.
Wiring it into GitHub or GitLab without slowing the pipeline down
Run the review step in parallel with your existing test suite, not before it, so it doesn't add to the critical path for merging a passing PR. Most teams trigger it on PR open and on each new push, with results posted as a single collapsed comment rather than individual line comments for every finding, which keeps the PR conversation readable. Set a timeout, typically two to three minutes is plenty for a scoped review, and fail open rather than closed: if the review step errors out, let the PR proceed rather than blocking merges on an infrastructure hiccup. Treat the review agent as an advisory layer that can go down without taking your whole delivery pipeline with it, the same way you'd treat an optional lint check rather than a required status check. Most CI providers make this easy to configure as a non-blocking job, so the plumbing cost is genuinely small compared to the review quality gained.
Choosing the three checks that matter for your codebase
The categories worth automating are the ones where a miss is expensive and a human reviewer is likely to be tired or rushed by the time they reach that part of the diff: permission and auth logic, anything touching money or billing, and database migrations that can't easily be rolled back. Teams that try to cover general code quality, naming, or style end up duplicating what a linter already does better and cheaper. Pick the two or three categories where getting it wrong actually costs the business a bad night, and leave the rest to your linter and your human reviewers.
Cost and where this doesn't help
Running an agent on every PR has a real per-run API cost, and on a busy repository with dozens of PRs a day that adds up, so budget for it rather than treating it as free. It also won't replace a human reviewer's judgement on architecture or whether a feature is the right one to build. It's a narrow, mechanical check layered on top of human review, not a substitute for it.
Automata AI sets up scoped CI review agents for Australian engineering teams, including the guardrails that keep comment volume useful rather than noisy. If your team's tried this and got ignored PR comments, contact us via /contact; the fix is usually narrowing scope, not abandoning the idea.
