The current crop of frontier open-weight models are sparse mixture-of-experts systems at a scale that changes how serving works entirely. Kimi K3 runs 2.8 trillion total parameters with 896 experts and 16 active per token. GLM-5.2 runs 744 billion parameters with 40 billion active. Understanding why these need dozens of accelerators rather than one saves Australian teams a lot of wasted planning, and a fair amount of wasted budget.
Sparse activation reduces compute, not memory
The appeal of a mixture-of-experts design is that only a fraction of parameters run per token. That cuts the arithmetic per forward pass substantially. It does not cut the memory requirement, because any expert might be selected for any token, so the full parameter set has to be resident somewhere fast.
At 2.8 trillion parameters, the numbers get uncomfortable quickly:
Weights alone consume several terabytes even at 8-bit precision.
No single accelerator holds that, so experts are sharded across many devices.
Every token's routing decision then becomes network traffic between those devices.
This is expert parallelism, and it means your interconnect becomes the bottleneck long before your compute does. A cluster with slow links between nodes will underperform a smaller cluster with fast ones, which is the opposite of what most procurement decisions optimise for.
The context window cost nobody quotes
A one million token context is the headline feature and the quiet expense. Key-value cache memory scales with context length, batch size and the number of attention layers. In practice, that plays out in three unglamorous ways:
Long contexts consume memory that could otherwise hold more concurrent requests.
Throughput per accelerator falls sharply as average context grows.
A deployment sized for 8k contexts will not survive users pasting 200k token documents.
Budget for context, not just for parameters. Teams routinely size a cluster from the model card and then discover their effective concurrency is a quarter of what they planned, usually right after the first Melbourne client uploads a full contract bundle instead of a summary.
What this means for an Australian deployment
If you are serving one of these models in an Australian region, the practical floor is dozens of accelerators, an engineer who understands the serving stack, and a monthly bill starting around $80,000 AUD. That is a reasonable commitment for a business selling inference as its product. It is not reasonable for a business that just wants better document handling for its own operations.
Mid-size open models, roughly 7 billion to 70 billion parameters, remain genuinely self-servable on one or two cards and are the right technical comparison for most Sydney and Brisbane teams evaluating a self-hosted option. For anything above that, a managed model such as Claude removes an infrastructure problem you would otherwise be solving permanently, and it sidesteps the APRA and Privacy Act questions that come with running a frontier-scale cluster yourself.
A quick gut-check before you size anything
Before committing budget to a self-hosted mixture-of-experts deployment, it's worth asking three plain questions: what's the actual average context length your users will send, not the marketing headline; what's your real concurrency requirement at peak, not average load; and does your team have someone who can debug an interconnect bottleneck at 2am, or would that page go to a vendor's support queue instead. Most Australian businesses that ask these questions honestly land on a managed model rather than a self-hosted cluster, and the ones that don't ask usually find out the hard way, a few months and a much larger invoice later.
Where the $80,000-a-month figure actually goes
It helps to unpack why the floor sits where it does rather than treating it as an arbitrary vendor number. Accelerator rental at scale, even at competitive cloud rates, accounts for most of it once you're running enough cards to hold a multi-terabyte weight set with headroom for KV cache. The remainder splits between interconnect bandwidth (the fast networking between nodes that expert parallelism depends on), storage for checkpoints and logs, and the engineering time to keep routing balanced so no single expert becomes a bottleneck under real traffic. None of these line items shrink much just because your actual usage is modest; the cluster has to be sized for the model, not for your workload.
Compare that to a Sydney business running the same document-heavy workload through a managed API. The equivalent monthly spend for meaningful volume, document processing, internal search, customer service triage, typically lands in the low thousands rather than the tens of thousands, because you're paying for tokens consumed rather than for a cluster sitting warm around the clock waiting for traffic.
If you want a serving architecture reviewed before you commit budget, book a session with Automata AI and we'll work through whether a self-hosted model actually clears the bar for your workload.
