Andrew Ng has published an updated skills map for what he calls AI engineering, and it works as a reality check for any Australian business now running pilots on Claude or another frontier model. Ng splits the discipline into six areas: LLM foundations, grounding models with data, agentic systems, evaluation-driven development, operating in production, and classical machine learning foundations.
Most of that list is not prompt writing. That is the part worth sitting with.
The evaluation habit is the differentiator
Ng's argument on evaluation is that the gap between engineers who ship reliable AI systems and everyone else comes down to a habit rather than a technique. Run the eval, do the error analysis, make the improvement, run the loop again. Traditional software lets you plan the build in advance because the behaviour is deterministic. AI systems produce uncertain output, so the only way to know whether a change helped is to test it against real cases and look closely at what broke.
That matches what we see inside Australian businesses moving from a Claude pilot to something that runs unattended. The teams that get stuck are usually the ones treating a working demonstration as the finish line. The teams that get a Claude Code or Cowork workflow into daily production are the ones who built the habit of testing outputs against real examples before trusting the system with real work.
The habit is cheap to start and almost impossible to retrofit. Twenty real cases, scored by a person who knows what good looks like, set up in an afternoon, is enough to make every subsequent change measurable. Skipping it saves that afternoon and costs months later, when nobody can say whether the system is getting better or just different.
Most of the map is systems engineering
The other detail worth noting is how much of Ng's map is agent infrastructure rather than model calls:
Tools and MCP connections, which determine what the agent can actually reach and what it cannot.
Memory and context management, which is where most long-session failures originate.
Agent harnesses and multi-agent coordination, including what happens when one step fails partway through.
Sandboxes, security and guardrails, which is the difference between an agent that can be trusted with a system of record and one that cannot.
None of that is prompt writing. It is closer to systems engineering, and it is exactly where the real work sits once a business moves past asking whether Claude can answer a question and into whether Claude can run a process every day without a human checking every step.
How to read an AI proposal against this map
For Australian business owners sizing up an automation project, the map is a decent gut check. If a proposal is entirely about the model and says nothing about evaluation, connections or failure handling, it is missing the part of the work that determines whether the system survives production.
Ask what working means in measurable terms, and what the number is today. A proposal that cannot answer this has not looked at your data.
Ask what happens when the agent gets something wrong. Silent failure is the expensive kind, and it is the default unless someone designed against it.
Ask which systems it connects to and how, because a demo with pasted data and a production system with live connections are different builds at different prices.
Ask who owns it after handover. An unowned automation degrades quietly, and the first sign is usually someone quietly going back to the spreadsheet.
A build that includes evaluation, connection work and failure handling costs more than one that does not. For a typical mid-market workflow that might be the difference between $12,000 and $35,000. The cheaper version is not a bargain if it stops being trusted in month three, and the honest way to compare two quotes is to check whether they are quoting the same scope rather than the same outcome.
The same test applies to vendors
This cuts both ways, and it should. Anyone in Australia selling Claude implementation work ought to be able to point at their own evaluation process rather than a demonstration. If a vendor cannot show what working means in measurable terms, or what happens when the model gets something wrong, that is the same gap Ng is describing wearing a different name tag.
Worth one caveat: the map describes a full discipline, and no mid-market business needs all six areas in-house. Classical machine learning foundations, for instance, rarely matter for a business using a frontier model through an API. The useful version is knowing which parts your project actually touches, and making sure somebody owns those, whether that is your team or ours.
If you have a Claude pilot that works in a demo and has not yet survived a week of real use, the missing piece is usually the evaluation loop rather than the model. book a session and we will look at what you have running now before proposing anything.
