The runtime decides more than the model does
Picking an open-weight model is only half the job for a business weighing a self-hosted AI setup. The runtime is the software layer that actually loads the model and serves it to your team, and it decides how fast responses come back, how many people can use it at once, and how much engineering effort it takes to keep the thing running. An Australian business that picks the wrong runtime can end up with a genuinely strong model that feels slow, drops requests under load, or breaks every time someone tries to update it.
Ollama: the simplest way to get a model running on a single machine, built for one developer or a very small team.
vLLM: designed for throughput, built to serve many people asking questions at the same time without falling over.
LocalAI: a drop-in replacement for the standard hosted-API shape, useful when you want to swap an open model in behind code you have already written.
Ollama: the fastest way to get something running
Ollama is where most teams start, and for good reason. It installs in minutes, pulls a model with a single command, and gives you a working local endpoint the same afternoon. For a Sydney developer testing whether an open model is even good enough for a task, or a two-person team running the occasional internal query, that speed matters more than raw capacity. The trade-off shows up once more than a handful of people try to use it at the same time. Ollama was not built for concurrent production traffic, and pushing it there usually means dropped requests, long queues, or a rebuild on a different runtime a few months later.
vLLM: built for concurrent production traffic
vLLM solves the problem Ollama was never designed for. Its request batching and memory management let a single GPU serve dozens of overlapping conversations instead of one at a time, which is what a support team fielding constant customer queries actually needs. The cost is operational complexity: you are running a proper service now, with containers, GPU drivers, load balancing, and monitoring to keep an eye on. A business adopting vLLM without someone who owns that infrastructure day to day is usually trading a model problem for a bigger reliability problem.
LocalAI: a drop-in replacement for the API you already wrote
LocalAI earns its place for a narrower reason: it mimics the standard hosted-API request and response shape, so software already written against that pattern can point at an open model with a small configuration change instead of a rewrite. That makes it the practical choice for a team that built an integration two years ago and now wants to test whether an open model can replace part of that spend without touching the application code. The trade-off is a smaller community and slower feature parity than Ollama or vLLM, so newer model formats and optimisations often land there last.
Matching the runtime to your traffic and team
For a proof of concept on one workstation, Ollama gets you running in an afternoon with no infrastructure to manage.
For production traffic across a team, vLLM handles concurrency far better, provided someone owns the operations.
For slotting an open model into software already built against a standard API, LocalAI saves the most rewriting.
What this actually costs an Australian business
The runtimes themselves are free. What they are not free of is the infrastructure underneath and the person who keeps it running, and that is where most self-hosting budgets go wrong.
A capable GPU server costs somewhere between $12,000 and $35,000 to buy outright, depending on whether you need one consumer-grade card or several data-centre GPUs.
Renting equivalent GPU capacity in an Australian cloud region typically runs $400 to $1,800 a month, scaling with model size and concurrent users.
Budget real engineering time to configure, secure, and monitor the runtime; even a few hours a week adds up to a meaningful slice of a $150,000-plus Australian engineering salary over a year.
Add a patching routine, since these tools ship security fixes often, and a runtime left unpatched for months becomes the actual risk, not the model running on it.
There is a genuine reason beyond cost that some Australian businesses choose to self-host anyway: data residency. A firm handling client data under the Privacy Act, or a business in a regulated sector keeping an eye on APRA expectations, may want model inference to happen inside a chosen Australian region or fully on-premises rather than sent to an overseas API. That is a legitimate driver for the runtime conversation. It is a much weaker reason if the only motivation is avoiding a per-token bill.
The maintenance question nobody budgets for
A runtime that is easy to install can still be hard to keep secure and current a year later. If nobody on the team owns that job explicitly, even the best runtime drifts out of date and turns into the risk you were trying to avoid by controlling your own infrastructure in the first place. Match the tool not just to today’s traffic, but to the maintenance time someone will genuinely commit to for the next two or three years.
When it is simpler to skip the runtime entirely
For most small and mid-sized Australian teams without a dedicated infrastructure engineer, a managed model like Claude avoids this entire layer: no GPU to buy, no patching schedule, no midnight page when the server falls over. That is why we default to a managed deployment unless a client has a specific, well-argued reason to self-host, usually data residency or a workload large enough that the unit economics genuinely favour owning the hardware. If you want a second opinion on which side of that line your business sits on, our Claude consultancy work starts with exactly that assessment before any infrastructure decision gets made.
If you are weighing a self-hosted runtime against a managed model, book a session and we will work through your actual traffic, team size, and compliance requirements before you commit to either path.
