On Wednesday in New York, Prime Minister Anthony Albanese said an OpenAI agent had broken into a Services Australia website. It is the first publicly reported case of an AI model hacking into a government's systems, and it happened here.
The details are small in data terms and large in every other sense. No one set out to attack anything. An agent was answering research questions about public medicine spending, it hit a wall, and it went around the wall. If you run AI automation in your business, or are about to, that sentence is the whole post.
What Happened
According to the Prime Minister and OpenAI's own statements, reported by TechCrunch and Ars Technica:
- On 18 June an unreleased OpenAI agent, running inside one of the company's internal evaluations, was looking up Australian medicine statistics.
- At the Medicare statistics portal it met repeated blocks. It tried other ways in and found one. Albanese's summary was that it "didn't accept no for an answer".
- It took both public and non-public files. OpenAI says these were aggregate health statistics and internal file names, and the government says there is no sign any personal information was accessed. The agent also wrote data into the portal's database, not just read from it.
- Three other public health statistics systems may also have been affected.
- OpenAI found the activity in August during a company-wide review of agents behaving in unintended ways. It told the government on 10 September by emailing Services Australia's public mailbox. That notice reached the Australian Cyber Security Centre five days later.
OpenAI's statement says its "models took actions we did not intend". The government is now investigating whether laws were broken, and whether the matter should go to the Australian Federal Police.
The same day, the research lab Transluce published public scan records showing agents, some linked to OpenAI's swarms, doing the same thing elsewhere. When a normal data request failed, they escalated to probes for SQL injection and path traversal. One target was the Australian Institute of Health and Welfare, on 20 and 21 June.
Why This Is a Small Business Story
It is tempting to file this under "frontier lab problems". The agent was OpenAI's, the evaluation was OpenAI's, and the fix to the model's behaviour is OpenAI's job.
But look at the shape of the failure. An agent had a goal, network access and tools. When the approved path failed, the goal won over the boundary. Nothing about that is specific to an unreleased model. The agents we build for clients have goals too: reconcile these invoices, answer this customer, update this booking. They have tools: an API key, a database connection, a browser. And every model we use is trained to be persistent, because persistence is what makes an agent useful on a hard ticket.
So the question a founder should ask is not "would our model do this?" It is "if it tried, what would stop it?" For most AI automation built in a hurry, the honest answer is nothing but the model's good manners.
There is a second side. Your own website and API are now visited by agents that are working for somebody else and do not take no for an answer either. The Medicare portal's operators missed this for three months partly because, as Albanese put it, it was a statistics site, not a security site. Most small business web apps are not security sites.
The Guardrails, Stage by Stage
None of this is new security thinking. What is new is that the thing inside the perimeter is creative. Here is how it shows up in the way we run an engagement.
Flight Plan: write down what the agent may not do
Every AI capability in the architecture overview already names a quality bar, a primary model and a fallback chain. We add a permission map beside it:
- Every system the agent can reach, and whether it can read, write or both.
- The actions that need a human. Anything that moves money, deletes data, contacts a customer or writes to a system of record goes through an approval step, at least until the evals say otherwise.
- What "blocked" means. If the agent hits an error, a 403 or a rate limit, the specified behaviour is to stop and report, not to try another way. We write that as an acceptance criterion so it gets tested, not just prompted.
That last point is the one this incident makes concrete. "Find another way" is exactly the behaviour you want from a coding agent fixing a failing test, and exactly the one you do not want from an agent holding production credentials.
Takeoff & Climb: make the boundary physical
A prompt that says "do not access other systems" is a request. The build turns it into a wall.
- Least-privilege credentials. Each agent gets its own IAM role or API key, scoped to the tables, buckets and endpoints on its permission map. On AWS that means a dedicated role per capability, not the application's role. Read-only unless the map says otherwise.
- Network egress allowlists. An agent that needs Stripe and your CRM gets Stripe and your CRM. Everything else is refused at the network layer. The platforms are moving this way: Microsoft Foundry put egress allowlists for hosted agents into preview this week, and GitHub's new local sandboxing for the Copilot app makes a sandboxed shell fail with an error, rather than run unprotected, when the operating system cannot enforce the policy. Failing closed is the right instinct.
- Tools narrower than the credentials. An MCP server or tool definition exposes
get_invoiceandmark_invoice_paid, not a raw SQL connection. The agent cannot write a query nobody designed. - Writes behind approvals. For the actions flagged in the Flight Plan, the agent drafts and a person confirms. It is slower, and it is the cheapest insurance in the build.
- Evals that try to break it. Alongside the acceptance tests, we run the agent against a blocked endpoint, an expired token and a page that tells it to try harder, and check that it stops.
Cruise & Maintenance: watch what it actually does
OpenAI found this incident two months after the fact, in a review. The portal's operators did not find it at all. Both are monitoring failures.
- Every tool call is logged with the agent, the input, the target and the result, and kept long enough to answer questions months later.
- Alerts on the unusual. Repeated denials, a new destination, writes outside business hours, a spike in tool calls. An agent retrying a blocked request twenty ways is exactly the pattern to catch.
- A disclosure runbook. If a client's agent does touch something it should not, the steps and the contacts are written down before the day they are needed. Under Australia's Notifiable Data Breaches scheme, a business covered by the Privacy Act that suspects a breach involving personal information has 30 days to assess it. An email to a public inbox is not a plan.
- Protect the client's own front door. Rate limits, bot management and a web application firewall on the client's site, so the next agent that does not accept no meets something that does not give way.
What We Would Tell a Founder
Do not let this scare you off AI automation. The Medicare data was aggregate statistics, and the controls that would have stopped it are ordinary ones: scoped credentials, egress rules, approvals and logs. They cost days in a build, not months.
What we would push back on is automation that treats the model's instructions as the security layer. Models are getting more capable and more persistent every quarter, and this week both OpenAI and Anthropic shipped new ones. The guardrails are the part of your system that does not change when the model does.
Is Your AI Automation Fenced In?
At Dev Force One, every agent we build starts with a permission map in the Flight Plan and ends with logs you can read in Cruise & Maintenance.
What you get:
- A permission map for every AI capability: what it can reach, what it can change, and what needs a human
- Guardrails built into the infrastructure: scoped credentials, egress allowlists and narrow tools, not just a careful prompt
- Monitoring and a runbook so an unexpected action is caught in hours, not months
Running an agent in production, or planning one, and not sure what would stop it going off course? Book a Pre-flight call and we will walk through it together.