A Claude Skill is a set of instructions, reference files and scripts that teaches Claude to do a job your way. Plenty of Australian teams now run a handful in production: a quoting skill, a report formatter, a compliance checklist. Far fewer have any way of knowing whether those skills still work after an edit. Testing Claude Skills sounds like something you need a machine learning team for. It is not. If your team can maintain a spreadsheet of examples, you can run a proper eval harness.
This guide walks through the harness we set up for consulting clients: what to measure, how to build golden cases, and how to grade outputs without touching a single ML library.
Why Skills Fail Quietly
Traditional software fails loudly. A broken function throws an error and a failed deploy pages someone. Skills fail differently. Claude still produces an answer, the answer still reads well, and nobody notices that the GST treatment is wrong or that the mandatory disclaimer dropped off. The output degrades instead of breaking, so the failure surfaces weeks later in front of a client rather than minutes later in a test run.
Three culprits account for most incidents: someone edits the skill and quietly breaks a case that used to work; a new model version shifts behaviour at the margins; or the skill stops triggering when it should, so Claude answers from general knowledge instead of your documented process. All three are cheap to detect and expensive to discover in production.
What a Skill Eval Actually Measures
An eval is just a list of realistic inputs paired with a definition of what good looks like. For skills, four checks cover most of the risk:
Triggering. Given a request that should invoke the skill, does Claude actually load it? Given a request that should not, does it stay out of the way? Both directions matter, and both drift as descriptions get edited.
Hard requirements. Facts and formats that are either right or wrong: the ABN is valid, the totals add up, the disclaimer is present, the output parses as JSON. A small script can score these deterministically.
Judgment quality. Tone, structure, whether the advice matches your firm's actual position. These need a grader with judgment, which nowadays can be another Claude instance working from a written rubric.
Edge cases. The odd inputs your team already knows about: the client with two trading names, the quote in NZD, the empty field. Every support ticket that traces back to the skill becomes a new eval case.
A Minimal Harness You Can Build in an Afternoon
Skip the frameworks to start. A workable harness is three folders and a script. A cases folder holds one file per test with the input prompt and the expected properties. A runs folder collects dated outputs. A runner script feeds each case to Claude with your skill loaded, then scores the result against the checks.
Start with ten golden cases pulled from real usage: five requests the skill handles daily, three edge cases lifted from support tickets, and two requests that should not trigger the skill at all. For each one, write down the deterministic checks (must contain X, must not contain Y, the number must equal Z) and a one-line description of what a good answer covers.
Run the whole set before you ship any change to the skill, the same way you would run unit tests before a deploy. A ten-case suite on Claude Haiku finishes in under two minutes, and the habit matters more than the coverage. Teams that run evals on every edit catch regressions the same day. Teams that rely on eyeballing catch them at quarter end.
Grading Without a Data Science Team
Deterministic checks score themselves. For the judgment checks, use a second Claude call as the judge: it receives the case input, your rubric and the output, and returns a pass or fail with a reason. Write the rubric the way you would brief a new hire: 'Pass if the response quotes the current award rate, states its assumptions, and recommends written confirmation. Fail if it invents rates or skips the confirmation step.'
Judges are not perfect, so anchor them. Include one known-good output and one known-bad output in the suite and confirm the judge scores both correctly before trusting it on new cases. When the judge and a human disagree, the human wins and the rubric gains a clarifying sentence. Within a few weeks the rubric becomes the most useful documentation the skill has.
What It Costs to Run
Very little, which is why the excuse budget is thin. A 25-case suite, run on every skill edit plus a weekly schedule, lands around $15 to $40 a month in API usage for most teams, judge calls included. Compare the downside: one Sydney professional services firm we worked with put the cost of a single incorrect client-facing figure, caught late, at roughly $18,000 in remediation time and difficult partner conversations. The harness paid for itself in its first week.
There is a compliance angle too. If your skills touch personal information, an eval suite doubles as evidence of reasonable steps under the Privacy Act: you can show exactly what the system was tested against and when. APRA-regulated firms will recognise the pattern; it is the same discipline as model risk controls, scaled down.
Start Smaller Than You Think
The failure mode to avoid is planning a 200-case suite and shipping nothing. Ten cases, one script and the discipline to run it before every edit beats a sophisticated framework nobody uses. Add a case every time something surprises you, and the suite grows into a real safety net within a quarter.
If you want help standing up an eval harness for the skills your team already runs, book a short call and we will map your first ten cases together.
