Blog

Claude and Distillation Attacks: Protecting Model Reasoning

November 2026 · 7 min read · Technical

Hand-drawn thought cloud with a terracotta padlock, and crossed-out cups beside it showing copied reasoning blocked
← Back to all posts

On 30 September 2026 OpenAI published an account of a coordinated campaign that tried to pull hidden reasoning out of its models at scale. Nobody broke encryption and nobody read stored conversations. The attackers simply asked questions, many thousands of times, and tried to coax the model into revealing the intermediate thinking it normally keeps behind the curtain.

For an Australian business choosing a model provider, the headline is less interesting than the mechanics. If you are building on Claude, or weighing Claude against another frontier model, this incident is a useful prompt to ask how each provider protects reasoning, and what that means for the intellectual property you put through their API.

What is a distillation attack on an AI model?

A distillation attack is an attempt to copy a model's behaviour by querying it in volume and training a cheaper model on the answers. When the target is reasoning, the attacker tries to extract the step-by-step working a model produces before its final answer, because that working is far more valuable training data than the answer alone.

The technique sits in a grey zone. Each individual request looks like ordinary API traffic. The harm only shows up in the pattern: unusual prompt templates, huge request counts, and many accounts working in concert. That is why detection depends on the provider watching aggregate behaviour, not on any single request being blocked.

What OpenAI says happened

Everything in this section is OpenAI's own account, from its disclosure. We have not independently verified it, and no third party had confirmed it as at November 2026.

  • Activity began on 1 July 2026, according to OpenAI.

  • It spiked on 24 and 25 July, with roughly 16,000 extraction attempts from more than 4,000 users at the peak.

  • OpenAI linked a related cluster of more than 15,000 users to the same effort.

  • OpenAI says the activity was fully halted by 28 July 2026.

OpenAI also attributes the core activity to individuals connected with another AI developer. That is an accusation, not an established fact, and naming the company adds nothing to the lesson for buyers, so we leave it out. The useful takeaway is that a determined, organised group spent weeks trying to harvest reasoning from a commercial API, and that the provider caught it through traffic analysis.

How Claude handles reasoning you never see

Claude's extended thinking works differently from a model that hides its reasoning entirely. Developers can switch thinking on through the API and set a budget for it. On current Claude models the API returns a summarised view of that thinking rather than the full raw trace, and some thinking can come back as encrypted blocks that Claude can verify on the next turn but the caller cannot read.

Two design choices matter here. First, a summary gives developers enough to debug and audit an agent without handing over a verbatim transcript that would make clean training data for a copycat. Second, encrypted blocks let multi-step agent work carry its reasoning forward between calls without exposing it in plain text. Neither is a guarantee against misuse, but both reduce what a mass-querying attacker can collect from any one response.

Anthropic, like other frontier labs, also prohibits using its outputs to train competing models in its commercial terms and runs abuse monitoring across accounts. The OpenAI case is a reminder that terms of service only work when someone is watching traffic closely enough to enforce them.

Why this matters to an Australian business building on Claude

Most of our clients are not worried about someone stealing a frontier model. They are worried about their own IP: the prompts, tool definitions, retrieval corpora and fine-grained business rules that make an internal agent useful. The same attack pattern applies at a smaller scale. If you expose a Claude-powered assistant to customers or partners, a motivated user can probe it repeatedly to reconstruct your system prompt, pricing logic or the documents behind its answers.

Where reasoning and IP can leak, and what to check before you deploy
Exposure pointWhat can leakPractical control
Public chat assistantSystem prompt, business rulesRate limits per user, prompt-leak testing before launch
Agent with tool accessTool schemas, internal API shapesLeast-privilege tools, server-side validation
RAG over internal docsSource passages verbatimDocument-level permissions, citation limits
Exposed reasoning or logsStep-by-step workingKeep thinking server-side, redact logs
Shared API keysUsage pattern, cost blowoutsPer-app keys, spend alerts in AUD

None of these controls is exotic. They are the same disciplines a security team already applies to any customer-facing web app. What changes with AI is that the attack surface is natural language, so testing has to include adversarial prompting rather than only scanning for code vulnerabilities.

A worked example

Take an illustrative Sydney insurance broker that deploys a Claude assistant to answer policy questions for clients. The assistant's value sits in a long system prompt encoding underwriting rules, plus a retrieval index of product disclosure statements. Building that might cost around $45,000 in consultant and staff time. Without per-user rate limits and leak testing, a competitor could open a handful of accounts and spend an afternoon asking the assistant to explain its own rules. With limits, monitoring and a prompt that refuses to restate its instructions, the same probing becomes slow, noisy and easy to spot.

Questions to put to any model provider

Use the OpenAI disclosure as a template for due diligence. Whichever provider you choose, ask:

  • Is raw reasoning ever returned through the API, or only a summary?

  • How does the provider detect coordinated extraction across many accounts?

  • What do the commercial terms say about training on outputs, and how is that enforced?

  • Where is data processed and retained, and how does that sit with the Privacy Act and, for regulated entities, APRA CPS 234?

  • Will the provider tell you if your account is caught up in an abuse investigation?

We cover the procurement side in detail in our guide to taking Claude Enterprise from security review to signed order form, and the wider capability comparison in Claude vs OpenAI on critical cyber capabilities. For how credentials are kept away from the model itself, see why connector tokens never enter the Cowork sandbox.

What not to conclude from this incident

It would be easy to read the disclosure as proof that one provider is less secure than another. That does not follow. OpenAI detected and stopped the campaign, which is the system working. The more honest conclusion is that every frontier provider faces this pressure, and the differences that matter to buyers are in design choices like how much reasoning is exposed, how quickly abuse is spotted, and how openly incidents are reported.

It also does not mean you should avoid exposing AI assistants to customers. It means you should treat them like any other public interface: assume someone will try to misuse it, and build the rate limits, logging and testing in from day one rather than after the first incident.

If you want a second opinion on a provider's security posture before you commit, our Claude services include a model-provider review for Australian teams, and the AI readiness assessment is a quick way to see where your gaps are. When you are ready to talk it through, book a call.

FAQ

Frequently asked questions

Can someone steal a Claude model by querying it?

Not the model itself. An attacker can query Claude in volume to imitate some of its behaviour, but Anthropic's terms prohibit training competing models on outputs and the API limits how much raw reasoning each response reveals.

Does Claude show its full reasoning through the API?

Claude's extended thinking can be enabled through the API, but current models return a summarised view of that thinking rather than the full raw trace, and some thinking can arrive as encrypted blocks the caller cannot read.

Was any customer data exposed in the OpenAI distillation campaign?

According to OpenAI's own disclosure, the campaign targeted hidden reasoning through normal queries. It was not an encryption breach and did not involve access to stored conversations, though this has not been independently verified.

How can a business stop users extracting its Claude system prompt?

Use per-user rate limits, test the assistant with adversarial prompts before launch, keep sensitive rules server-side where possible, and monitor logs for repeated probing of the assistant's instructions or source documents.

Ready to move from AI pilot to production?

We help mid-market Australian businesses deploy AI automations that actually reach production and deliver measurable ROI.