On 30 September 2026 OpenAI published an account of a coordinated campaign that tried to pull hidden reasoning out of its models at scale. Nobody broke encryption and nobody read stored conversations. The attackers simply asked questions, many thousands of times, and tried to coax the model into revealing the intermediate thinking it normally keeps behind the curtain.
For an Australian business choosing a model provider, the headline is less interesting than the mechanics. If you are building on Claude, or weighing Claude against another frontier model, this incident is a useful prompt to ask how each provider protects reasoning, and what that means for the intellectual property you put through their API.
What is a distillation attack on an AI model?
A distillation attack is an attempt to copy a model's behaviour by querying it in volume and training a cheaper model on the answers. When the target is reasoning, the attacker tries to extract the step-by-step working a model produces before its final answer, because that working is far more valuable training data than the answer alone.
The technique sits in a grey zone. Each individual request looks like ordinary API traffic. The harm only shows up in the pattern: unusual prompt templates, huge request counts, and many accounts working in concert. That is why detection depends on the provider watching aggregate behaviour, not on any single request being blocked.
What OpenAI says happened
Everything in this section is OpenAI's own account, from its disclosure. We have not independently verified it, and no third party had confirmed it as at November 2026.
Activity began on 1 July 2026, according to OpenAI.
It spiked on 24 and 25 July, with roughly 16,000 extraction attempts from more than 4,000 users at the peak.
OpenAI linked a related cluster of more than 15,000 users to the same effort.
OpenAI says the activity was fully halted by 28 July 2026.
OpenAI also attributes the core activity to individuals connected with another AI developer. That is an accusation, not an established fact, and naming the company adds nothing to the lesson for buyers, so we leave it out. The useful takeaway is that a determined, organised group spent weeks trying to harvest reasoning from a commercial API, and that the provider caught it through traffic analysis.
How Claude handles reasoning you never see
Claude's extended thinking works differently from a model that hides its reasoning entirely. Developers can switch thinking on through the API and set a budget for it. On current Claude models the API returns a summarised view of that thinking rather than the full raw trace, and some thinking can come back as encrypted blocks that Claude can verify on the next turn but the caller cannot read.
Two design choices matter here. First, a summary gives developers enough to debug and audit an agent without handing over a verbatim transcript that would make clean training data for a copycat. Second, encrypted blocks let multi-step agent work carry its reasoning forward between calls without exposing it in plain text. Neither is a guarantee against misuse, but both reduce what a mass-querying attacker can collect from any one response.
Anthropic, like other frontier labs, also prohibits using its outputs to train competing models in its commercial terms and runs abuse monitoring across accounts. The OpenAI case is a reminder that terms of service only work when someone is watching traffic closely enough to enforce them.
Why this matters to an Australian business building on Claude
Most of our clients are not worried about someone stealing a frontier model. They are worried about their own IP: the prompts, tool definitions, retrieval corpora and fine-grained business rules that make an internal agent useful. The same attack pattern applies at a smaller scale. If you expose a Claude-powered assistant to customers or partners, a motivated user can probe it repeatedly to reconstruct your system prompt, pricing logic or the documents behind its answers.
| Exposure point | What can leak | Practical control |
|---|---|---|
| Public chat assistant | System prompt, business rules | Rate limits per user, prompt-leak testing before launch |
| Agent with tool access | Tool schemas, internal API shapes | Least-privilege tools, server-side validation |
| RAG over internal docs | Source passages verbatim | Document-level permissions, citation limits |
| Exposed reasoning or logs | Step-by-step working | Keep thinking server-side, redact logs |
| Shared API keys | Usage pattern, cost blowouts | Per-app keys, spend alerts in AUD |
None of these controls is exotic. They are the same disciplines a security team already applies to any customer-facing web app. What changes with AI is that the attack surface is natural language, so testing has to include adversarial prompting rather than only scanning for code vulnerabilities.
A worked example
Take an illustrative Sydney insurance broker that deploys a Claude assistant to answer policy questions for clients. The assistant's value sits in a long system prompt encoding underwriting rules, plus a retrieval index of product disclosure statements. Building that might cost around $45,000 in consultant and staff time. Without per-user rate limits and leak testing, a competitor could open a handful of accounts and spend an afternoon asking the assistant to explain its own rules. With limits, monitoring and a prompt that refuses to restate its instructions, the same probing becomes slow, noisy and easy to spot.
Questions to put to any model provider
Use the OpenAI disclosure as a template for due diligence. Whichever provider you choose, ask:
Is raw reasoning ever returned through the API, or only a summary?
How does the provider detect coordinated extraction across many accounts?
What do the commercial terms say about training on outputs, and how is that enforced?
Where is data processed and retained, and how does that sit with the Privacy Act and, for regulated entities, APRA CPS 234?
Will the provider tell you if your account is caught up in an abuse investigation?
We cover the procurement side in detail in our guide to taking Claude Enterprise from security review to signed order form, and the wider capability comparison in Claude vs OpenAI on critical cyber capabilities. For how credentials are kept away from the model itself, see why connector tokens never enter the Cowork sandbox.
What not to conclude from this incident
It would be easy to read the disclosure as proof that one provider is less secure than another. That does not follow. OpenAI detected and stopped the campaign, which is the system working. The more honest conclusion is that every frontier provider faces this pressure, and the differences that matter to buyers are in design choices like how much reasoning is exposed, how quickly abuse is spotted, and how openly incidents are reported.
It also does not mean you should avoid exposing AI assistants to customers. It means you should treat them like any other public interface: assume someone will try to misuse it, and build the rate limits, logging and testing in from day one rather than after the first incident.
If you want a second opinion on a provider's security posture before you commit, our Claude services include a model-provider review for Australian teams, and the AI readiness assessment is a quick way to see where your gaps are. When you are ready to talk it through, book a call.



