Open-weight AI vendor pitches have converged on a familiar shape this year. A benchmark chart showing near-parity with frontier models. A cost comparison that looks dramatically cheaper than your current platform. And a claim that switching your existing stack over is straightforward.
Some of that is true. Independent research this year found open-weight models now cover roughly 80% of proprietary use cases at meaningfully lower cost. What the pitch usually leaves out is where the other costs live, and Australian businesses keep getting caught by the same handful of gaps.
The five questions worth asking before you sign
What does the licence actually restrict, in writing rather than in the sales deck? Licences shift. MiniMax moved one of its models from a permissive MIT licence to a non-commercial one requiring written authorisation, with no obligation to grandfather existing users.
Has an independent party tested this model's safety behaviour, or only the publisher? Some open-weight models tested this year refused none of the offensive or dual-use prompts researchers put to them, which is a different risk profile to the one the benchmark chart describes.
What is the fully loaded cost, including the people? Industry research suggests 85% of organisations misjudge AI project costs by more than 10%, typically because the sticker price excludes infrastructure, engineering time and ongoing maintenance.
Where does the data actually sit, and under whose legal jurisdiction? Self-hosted is not automatically sovereign if the infrastructure or the parent company sits under a foreign jurisdiction, regardless of where the servers physically are.
What happens if you need to walk away in twelve months? Ask whether you can export everything you have built, or whether leaving means rebuilding your integration layer from nothing.
Why these five and not others
Each one maps to a failure we have watched happen rather than a theoretical risk. The licence question catches terms changing under you. The safety question catches a model that behaves differently in production than in a demonstration. The cost question catches the labour nobody budgeted. The jurisdiction question catches a data sovereignty claim that does not survive a lawyer reading it. And the exit question catches the lock-in that open-weight pitches implicitly promise to avoid and frequently reproduce.
Note what is not on the list: benchmark performance. Not because capability does not matter, but because it is the one thing every pitch already covers thoroughly, and the one thing you can verify yourself in an afternoon against your own tasks. The five above are the ones you can only get answers to by asking, and the answers are much harder to reconstruct after you have signed.
What this checklist has caught in practice
We have run this exact set against vendor pitches for Australian clients three times this year, and twice it changed the recommendation.
One case saved a client from a $35,000 commitment to a model whose licence would have made their planned resale use illegal, a detail nobody in the room had spotted because the deck said open and everyone read that as unrestricted. The other surfaced a support gap that would have left the client with no safety net if the vendor's small team could not keep up with a production incident, which is a reasonable risk to take knowingly and a poor one to discover during the incident.
How to actually use it
Send the five questions before the meeting rather than asking in it. Written answers are better answers, and you find out quickly how a vendor handles scrutiny.
Get the licence answer from whoever reads your contracts, not from your engineering team. It is not an engineering question.
Ask the cost question with your own volume attached, so the answer is a number rather than a rate card.
Treat a vague answer as an answer. A vendor who cannot say where the data sits has told you something useful.
None of this is an argument against open-weight models. It is an argument against taking a benchmark chart at face value, and it applies just as much to a managed platform pitch as an open-weight one. Claude does not automatically win all five of these questions. What it does is answer all five clearly before you sign, and that clarity is the actual comparison worth making.
If you have a vendor pitch on your desk and want the five questions run against it properly, with the answers written down where a board can read them, book a session and we will go through it with you.
