A voice agent that can hold a real phone conversation used to be a call-centre project with a six-figure budget and a nine-month timeline. That has changed. With Claude as the reasoning layer, Vapi handling the telephony, and ElevenLabs generating the voice, a small Australian team can stand up a working phone agent in days rather than quarters. This guide walks through what the stack does, what it costs, and where these agents tend to fall over.
What a Claude voice agent actually is
A voice agent is not a single product. It is three jobs stitched together: turning speech into text, deciding what to say, and turning text back into speech. Claude does the middle job, which is the hard one. It reads the caller's transcribed words, checks any tools or knowledge it has been given, and decides on a reply that stays on-topic and on-brand.
For a Sydney property advisory firm we work with, the agent answers inbound enquiry calls, qualifies the caller against a short set of questions, and books a discovery call straight into the calendar. The caller experiences one smooth conversation. Under the hood, three services are passing data back and forth several times a second.
The pieces you are assembling are:
Speech-to-text (transcription): converts the caller's audio into words Claude can read. Vapi manages this for you.
The reasoning model: Claude reads the transcript, applies your instructions, calls tools if needed, and writes the reply.
Text-to-speech: ElevenLabs turns Claude's reply into natural audio played back down the line.
The orchestrator: Vapi ties the three together, handles the phone number, interruptions, and call state.
The stack: Vapi, Claude, and ElevenLabs
Vapi is the telephony layer. It owns the phone number, answers the call, streams audio, and manages turn-taking so the caller can interrupt the agent mid-sentence without the conversation breaking. You configure it with a system prompt, a chosen model, a chosen voice, and a set of tools the agent is allowed to call.
Claude is the brain. We reach for Claude Sonnet for most voice work because latency matters on a live call and Sonnet answers fast while still following multi-step instructions reliably. The system prompt is where most of the engineering happens: you tell Claude who it is, what it can and cannot promise, how to handle an unclear answer, and when to hand off to a human. A tool definition lets Claude check a calendar, look up an order, or write a booking, all while the caller waits a beat.
ElevenLabs supplies the voice. Voice choice matters more than teams expect. A warm, natural voice buys patience from callers; a robotic one gets hung up on. For Australian callers we test voices for accent neutrality, because a strong American cadence reads as a spam call to a lot of people here.
The flow on every turn looks like this: the caller speaks, Vapi transcribes, Claude reads the running transcript and replies, ElevenLabs speaks the reply, and the loop repeats until someone hangs up or a tool books the outcome.
What it costs to run
The honest answer is that a voice agent is cheap to build and metered to run. There is no large upfront licence. You pay per minute of conversation across the three services plus the phone number.
As a working estimate, a typical business call runs the Vapi platform fee, the Claude tokens for that call, and the ElevenLabs audio to somewhere around $0.30 to $0.60 per minute all-in. A five-minute call therefore costs roughly $1.50 to $3.00. A firm handling 1,000 calls a month at an average of four minutes is looking at something in the range of $1,500 to $2,500 per month in usage.
Set that against the alternative. A part-time receptionist to cover the same after-hours enquiry volume costs a small business well over $45,000 a year once you include on-costs, and cannot answer three calls at once. The build itself, if you bring in help, is a fixed project rather than a headcount. The point is not that the agent replaces a person, but that it catches the calls a person was never going to reach.
Budget for a pilot before you commit. We advise clients to run a paid pilot on real calls for a month, measure booking rate and drop-off, and only then decide whether to widen the deployment.
Where these agents break, and how to keep them working
Most voice-agent failures are not model failures. They are configuration and edge-case failures. The recurring ones we see:
Dead ends on booking. The agent qualifies the caller beautifully then has no tool wired up to actually book, so the call fizzles. Confirm the booking tool exists and is tested before launch.
Wrong voice, wrong region. A default American voice on an Australian line lowers trust. Pick and test the voice deliberately.
No human handoff. Callers with an unusual request need an exit. Give the agent a clear rule for when to transfer or take a message.
Silence handling. If the caller pauses, a badly tuned agent talks over them or gives up. Endpointing settings need tuning against real recordings.
Prompt leakage. Without guardrails, an agent can be talked into revealing its instructions or going off-script. Constrain scope in the system prompt and test adversarial callers.
The fix for almost all of these is the same discipline: record real calls, review the transcripts weekly, and tighten the prompt and tools against what actually went wrong. A voice agent is a system you maintain, not a thing you ship once.
Getting started
If you are scoping your first agent, start narrow. Pick one call type with a clear outcome, such as booking a callback or answering the three questions you get asked most. Write the system prompt as if you were training a new junior on their first shift: who they are, what they can promise, and what to do when they are unsure. Wire up exactly one tool that produces the outcome, and test it on your own phone before anyone else calls.
Under Australian Privacy Act obligations you should also tell callers they are speaking with an automated assistant and that the call may be recorded, and store any personal information you capture accordingly. Building that disclosure into the opening line is straightforward and keeps you on the right side of the rules.
We build and tune Claude voice agents for Australian businesses end to end, from the first prompt to a monitored production line. If you want to talk through whether a voice agent fits your call volume, book a brainstorm with us and we will map it out with you.
