Every few months a new piece of AI hardware makes the headlines. The latest is a reported custom inference chip, nicknamed 'Jalapeno', that OpenAI is said to be building with Broadcom. The details are still second-hand and worth treating with caution, but the story underneath matters for any Australian business budgeting for AI. The cost of running models, not just training them, is becoming the main event.
For most Sydney and Melbourne teams using Claude day to day, a chip roadmap at a US lab sounds remote. It is closer than it looks. Inference is the compute that happens every time a model answers a prompt, and inference is what shows up on your monthly bill. When the large labs pour money into custom silicon, they are chasing cheaper inference at scale, and that pressure eventually reaches the prices you pay.
Why inference cost is the number that matters
Training a model is a one-off, headline-grabbing expense for the lab that builds it. Inference is different. It is the recurring cost of actually using the model, and for a business it behaves like electricity: you pay for it every time you switch something on. A model you call a thousand times a day is an operating cost, not a capital purchase, which is why a small change in the per-token price can move your bill more than any single feature.
A team running Claude across customer support, drafting, and analysis might spend somewhere between $2,000 and $8,000 a month on inference depending on volume. Halve the per-token cost and that line halves with it. That is the real reason the compute arms race should be on your radar, even if custom chips are not.
Volume: how many requests you send and how long each prompt is. High-traffic automations cost more than occasional use.
Model choice: a larger model costs more per call. Many everyday jobs run perfectly well on a smaller, cheaper one.
Context size: a 200,000-token prompt costs far more than a 5,000-token one. Sending an entire document when a summary would do is money out the door.
Reuse: repeated context can often be cached rather than re-sent, which trims the bill on workflows that run the same setup again and again.
What the 'Jalapeno' reports actually signal
Treat the specifics as unconfirmed. The reporting points to a custom inference chip aimed at driving down the cost of serving models, but the timeline, performance, and even the name could change. What is clear is the direction of travel. Every major lab wants cheaper inference, because inference is where the volume, and the margin, lives.
Here is where Claude is relevant for an Australian buyer. Claude already runs across more than one cloud platform, which means you are not tied to a single provider's hardware or pricing. If one platform becomes cheaper or faster, you have room to move. The lesson from the chip race is not to bet on a particular piece of silicon. It is to keep your AI setup portable and your costs measured, so you can take advantage of price drops whoever delivers them.
What an Australian business should do now
Start by measuring what you actually spend on inference today, broken down by workflow. Most teams have never looked, and the first audit usually finds a handful of jobs quietly running on an oversized model or sending far more context than they need. Right-size the model to the task, cap the context where you can, and revisit the numbers each quarter as prices move.
Keep the Australian context in view too. Where inference runs has privacy and data-residency implications under the Privacy Act, and for regulated firms the location of processing is not just a cost question. A sensible setup balances price against where your data is allowed to go, rather than chasing the cheapest option on a US price list.
A simple way to think about the trade-off
It helps to picture inference spend as three dials rather than a single number. The first dial is how often you call the model, the second is how large each call is, and the third is which model answers. Most teams reach for the third dial first by swapping models, when the easier wins sit on the first two. Trimming a bloated prompt or batching repetitive requests often saves more than a model change, and it carries none of the quality risk that comes from dropping to a weaker model for work that needs the stronger one.
The compute race will keep pushing the per-token price down over time, which is good news for buyers. A falling unit price does not help much, though, if your usage is climbing faster and nobody is watching those dials. The businesses that come out ahead treat AI spend like any other operating cost, with a named owner, a monthly figure, and a quarterly review, rather than a mystery line that only earns attention when it suddenly jumps. None of this requires deep technical knowledge. It requires someone to look at the numbers on a regular basis and ask whether each workflow is using more than it needs.
If you want a clear view of your AI compute costs and a plan to keep them down as the market shifts, we can help. You can book a brainstorm and we will map your current spend and where it can safely come down.
