Blog

Customer Triage: Off-the-Shelf or Fine-Tuned Model?

November 2026 · 7 min read · ROI & Business Case

Hand-drawn balance scale weighing a single box against a stack of labelled cards
← Back to all posts

Open decision models have given Australian businesses a choice for customer triage that barely existed six months ago. Cloudflare's Clef-flash and Amazon's Strands Decider can sort support tickets, route enquiries and score leads straight out of the box. You can also take either one and fine-tune it on your own historical decisions. Both routes are now realistic for a mid-market firm. They are not the same decision, and picking the wrong one costs either money or accuracy.

This post covers the build-versus-buy call for the classify step only. Whichever path you take, the cases that need real judgment still go to Claude. What changes is how well the triage layer in front of it handles your particular mix of work.

Should you use an off-the-shelf decision model or fine-tune your own classifier?

Use an off-the-shelf decision model when your triage categories are fairly standard, your volume is modest and you need something running this quarter. Fine-tune when a misclassification is expensive, when your categories are specific to your product or regulatory context, or when volume is high enough that a few points of accuracy pay back the extra build. Volume and error cost decide it, not preference.

The rest of this post puts numbers and signals against each of those conditions so you can place your own business on the map.

The off-the-shelf path

Running Clef-flash or Strands Decider as released, with prompt-based classification and no fine-tuning, is the fast option. A working triage layer is typically one to two weeks of engineering away. Model costs are close to negligible: Strands Decider is free to self-host, and Clef-flash runs at $0.09 per million input tokens on Cloudflare's hosted endpoint. The real spend is integration, typically $6,000 to $12,000 in AU contractor time for a first working pipeline.

Accuracy on your edge cases will be whatever a general-purpose decision model achieves on your prompts. That is usually fine for clearly defined categories and weaker on the ambiguous cases unique to your business. This path suits you right now if:

  • You need triage running this quarter and can live with a few months of imperfect edge-case handling while you collect data.

  • Your categories are standard, such as urgent versus routine or sales versus support versus complaint, rather than specific to your product.

  • You do not yet have a clean, labelled set of past triage decisions to train on, even if you wanted to.

The fine-tuned path

Fine-tuning either model on your own labelled tickets or leads is slower and dearer, but it closes the accuracy gap on the cases that matter to you. A realistic AU budget is $15,000 to $35,000, covering labelling of 2,000 to 5,000 good examples, the fine-tuning run and validation. The timeline is four to eight weeks rather than one to two.

Most of that budget goes on labelling, not compute. Someone who understands your categories has to look at thousands of past cases and record the right answer. That work also leaves you with a test set you can reuse every time you change the model, which has value well beyond this one project. For the wider trade-offs between prompting, retrieval and fine-tuning, see our cost-quality triangle explainer.

Off-the-shelf versus fine-tuned decision model for customer triage, AUD, November 2026
FactorOff-the-shelfFine-tuned
Upfront cost$6,000 to $12,000$15,000 to $35,000
Time to first working version1 to 2 weeks4 to 8 weeks
Labelled data neededNone to start2,000 to 5,000 examples
Accuracy on standard categoriesUsually fineUsually fine
Accuracy on unusual categoriesWeakerMuch stronger
Best fitUnder about 10,000 decisions a monthHigh volume or costly errors

Three tests that decide it

Test one: monthly volume

If you process fewer than roughly 10,000 triage decisions a month, the off-the-shelf path almost always wins on cost. The fine-tuning spend will not pay back in a sensible timeframe at that volume, because there are not enough decisions for a few points of accuracy to add up to real money.

Test two: the cost of a wrong call

If a misclassification is expensive, fine-tuning can earn its keep even at lower volume. A mis-routed enquiry with compliance implications, a hardship request that sits in the wrong queue for a week or a high-value lead marked as low priority can each cost more than the whole labelling budget. In those cases the accuracy gain is doing real financial work.

Test three: how unusual your categories are

If your labels go beyond urgent versus not urgent and reflect something specific to your product or your regulator, an off-the-shelf model will underperform whatever your volume. A general model has never seen your internal taxonomy. Fine-tuning becomes the only realistic route to usable accuracy, and the first two tests matter less.

A worked example from Brisbane

A Brisbane services business running 25,000 tickets a month compared both paths. The off-the-shelf deployment came in at roughly $8,000 upfront. Fine-tuning came in at roughly $22,000 upfront, but it lifted triage accuracy on the firm's unusual categories from around 78% to around 93%. At that volume, fewer mis-routed complaints and less manual re-triage paid back the $14,000 gap within the first year.

To see why, run the illustrative arithmetic. A 15-point accuracy lift on 25,000 tickets is about 3,750 fewer misroutes a month, or around 45,000 a year. Recovering $14,000 over that many tickets needs each avoided misroute to save only about 31 cents in staff time. A single manual re-triage takes longer than that, so the case closes comfortably. Run the same sums at 3,000 tickets a month and the saving per avoided misroute has to be several times larger, which is where the off-the-shelf path usually wins.

A sensible sequence for most firms

For many mid-market teams the answer is both, in order. Start off the shelf, log every triage decision along with what a person later changed, and let that log become your labelled set. After a few months you will know your real volume, your real error rate and which categories the general model struggles with. You can then decide on fine-tuning with evidence rather than a forecast. Our broader build-versus-buy framework applies the same staged thinking to other AI tools, and our guide to pairing a decision model with Claude covers the routing design either path plugs into.

Either way, the decision model only replaces the parts of your pipeline where no reasoning was happening anyway. Complaints, ambiguous requests and anything needing judgment still go to Claude. If you want a straight answer on which path fits your volume and error profile, send us a sample of your triage data and we will tell you, including when the cheaper option is the right one.

FAQ

Frequently asked questions

How many labelled examples do I need to fine-tune a triage model?

Plan on 2,000 to 5,000 well-labelled examples drawn from your own past decisions. Quality matters more than raw count, so the labels should come from staff who know your categories well.

Is 10,000 decisions a month a hard cut-off for fine-tuning?

No. It is a rough guide for when accuracy gains start to pay back on volume alone. Expensive errors or unusual categories can justify fine-tuning well below that level.

Can I start off the shelf and fine-tune later?

Yes, and for many firms that is the best order. Log every decision and every staff correction from day one, and that log becomes the labelled data a later fine-tuning project needs.

Does a fine-tuned decision model replace Claude?

No. It handles classification only. Cases needing judgment, careful drafting or multi-step reasoning should still be routed to Claude, with a person involved where the stakes are high.

Ready to move from AI pilot to production?

We help mid-market Australian businesses deploy AI automations that actually reach production and deliver measurable ROI.