The Notifiable Data Breaches (NDB) scheme has been part of the Privacy Act 1988 since 2018, and most Australian businesses above the $3 million turnover threshold already know the shape of it: if personal information is lost or accessed in a way likely to cause serious harm, you assess the incident, and if it qualifies you notify the affected people and the Office of the Australian Information Commissioner (OAIC). What has shifted is the number of places that data now travels. Staff paste text into chat tools, vendors run models over your records, and outputs sometimes contain information no one intended to expose. This post walks through how the NDB scheme applies once AI tools are in the picture, and what a sensible response looks like.
What the NDB scheme actually requires
The scheme turns on one question: is this an eligible data breach? Three conditions have to line up.
There is unauthorised access to, unauthorised disclosure of, or loss of personal information your organisation holds.
A reasonable person would conclude the breach is likely to result in serious harm to one or more individuals.
You have not been able to prevent that harm through remedial action.
If all three hold, you have 30 days from becoming aware of the incident to complete your assessment, and you must notify affected individuals and the OAIC as soon as practicable. The stakes are not small. Since late 2022, serious or repeated privacy breaches can attract civil penalties of up to $50 million, and the average cost of a data breach in Australia now sits at roughly $4.26 million once remediation, downtime and lost trust are counted. AI tools do not create a new legal test. They create new ways for the existing test to be met.
Where AI tools change the breach picture
The awkward part of AI-related breaches is that they rarely look like a classic hack. Nobody breaks through a firewall. Information simply ends up somewhere it should not be. Three scenarios cover most of what we see with Australian clients.
A staff member pastes client data into an unapproved tool
An account manager copies a spreadsheet of customer names, emails and payment histories into a consumer chatbot to draft a mail-out. That tool retains inputs for training, and its terms grant the vendor broad reuse rights. The data has now been disclosed to a third party without authorisation. Whether it counts as an eligible breach depends on the sensitivity of the fields and how the vendor handles them, but the disclosure has already happened. This is the most common AI incident, and it is almost always avoidable with a clear tool policy and a private, no-training deployment such as Claude through a managed API or enterprise plan.
Your AI vendor is breached
You did everything right, but the provider holding your prompts and outputs suffers an incident of its own. If your data was in their systems, their breach can become your notification obligation, because the NDB scheme follows the information rather than the server. This is why contract and data-handling terms matter as much as model quality. Knowing where prompts are stored, for how long, in which country, and whether they are used for training is part of due diligence, not paperwork for later.
The model output leaks someone else's data
A support agent asks an internal assistant a question and the answer includes another customer's details, pulled from a poorly scoped knowledge base or a shared context window. The individual whose data surfaced never consented to that exposure. Retrieval design and access controls, not the model itself, are usually the cause, and they are usually fixable once you know where the boundary broke.
A response playbook
When an AI-related incident surfaces, the clock and the paperwork are the same as for any other breach. A workable sequence:
Contain first. Revoke the tool's access, disable the integration, or pull the affected dataset before anything else.
Preserve evidence. Capture prompts, outputs, logs and timestamps; you will need them for the assessment and any OAIC engagement.
Assess against the three conditions. Decide whether serious harm is likely, and document the reasoning even if you conclude it is not notifiable.
Notify if required. Tell affected individuals and the OAIC as soon as practicable, with clear guidance on what they should do next.
Fix the pathway. Close the specific gap that allowed the exposure, not just the single instance.
Firms in regulated sectors carry extra weight here. A financial services business also answers to APRA's CPS 234 expectations, and any AUSTRAC-reporting entity has to think about how a breach interacts with its own obligations. The NDB assessment does not replace those duties; it sits alongside them.
Lowering the odds before anything goes wrong
Most AI breaches trace back to two gaps: staff using tools no one approved, and data flowing into services whose terms nobody read. Both are governance problems, not technology problems. A short approved-tools list, a private deployment of Claude that does not train on your inputs, tight retrieval scoping, and a half-day of staff training will remove the large majority of the risk. The aim is not to slow people down. It is to give them a safe, fast option so they stop reaching for the unsafe one.
If you want a clear view of where client data currently flows through the AI tools your team uses, and a plan to close the gaps before they turn into notifications, we can help. Book a brainstorm and we will map it with you.
