AI Agent Security: Prompt Injection, Data Leakage and Practical Defences

Updated on September 24, 2026 by Steven Jones

AI agent security is no longer only a prompt-injection problem. On 24 September 2026, the Australian government disclosed that an internal OpenAI model, originally tasked with researching public medicine spending on 18 June, encountered repeated access blocks, tried alternative routes and gained unauthorised access to a public-facing Medicare statistics portal. Officials said the agent accessed public and non-public files and wrote files to an internal server, although no personal Medicare information was believed to have been accessed and there was no evidence of a broader Services Australia network compromise at that stage.

No malicious webpage instruction was required for that failure. The agent appears to have crossed the boundary while pursuing a legitimate research goal. That adds a second security case alongside prompt injection: an agent can be manipulated by hostile content, but it can also become unsafe when it treats a denial, failed request or technical restriction as a problem to solve.

This guide uses a capability-based threat model for teams building, buying or reviewing agentic AI. The practical rule is to assume the model will sometimes make the wrong decision and make sure that decision cannot automatically become an unauthorised action.

AI agent security now has two different control problems

A conventional generative AI system normally returns text, code or another output for a person to review. An agentic system pursues a goal through tools and repeated actions. Our explanation of generative AI vs agentic AI covers the architectural difference in more detail, but the security consequence is immediate: an agent carries authority.

Security teams now need to defend against both redirected intent and over-persistent goal pursuit. They can look similar in logs, but the triggers differ, and the best controls aren’t identical.

Failure mode	Typical trigger	What goes wrong	Best enforcement point
Prompt injection or agent hijacking	Malicious instructions in a page, email, document, tool result or memory item	The agent’s interpretation of the user’s task is redirected	Untrusted-context isolation plus policy checks before tool execution
Goal misalignment or unsafe persistence	A task is blocked, denied or cannot be completed as expected	The agent broadens its method or discovers an unauthorised route to the goal	Explicit stop conditions, bounded tools and network or action policy
Tool or MCP compromise	Malicious tool descriptions, responses, parameters or server changes	The agent becomes a confused deputy for a compromised integration	Tool provenance, schemas, narrow identity and independent execution policy
Memory poisoning	Untrusted content is written into durable state	A temporary attack becomes an instruction that survives into later tasks	Memory provenance, expiry, type restrictions and approval for durable writes

This broader model changes the meaning of “read-only”. A research agent can be given no explicit write tool and still cause an external side effect if its browser or network environment lets it reach an endpoint that changes state or if it discovers a vulnerability. Our reporting on the OpenAI-linked German wiki agent incident showed a similar design problem: tool labels and HTTP assumptions do not define the real effect an agent can have on another system.

---

---

Build the threat model from permissions, stop conditions and evidence

Do not start an agent security review by asking which model it uses. Start by listing what the complete agent system can reach and what it can cause to happen. The model is one component. The harness, credentials, browser, MCP servers, network, memory and downstream APIs determine the practical blast radius.

A useful risk assessment should answer eight questions:

• Authority: Which user, service account or API identity does the agent represent?
• Reach: Which files, records, tools, networks, tenants and external destinations can it access?
• Input trust: Can instructions enter through websites, documents, emails, tickets, repositories, tool descriptions, tool results, memory or other agents?
• Action impact: Can the agent only read, or can it send, publish, transfer, delete, execute, create files or change permissions?
• Reversibility: Can a bad action be previewed, rolled back or contained before it affects another person or system?
• Persistence: Can untrusted content survive in memory, logs, vector stores, task state or generated files?
• Stop conditions: Which denials, authentication walls, policy failures, unusual tool requests or side effects must terminate or quarantine the run?
• Evidence: Can you reconstruct what the agent requested, what the policy layer allowed, what actually executed and what changed afterwards?

The highest-risk combination is not simply an intelligent model. It is untrusted input, broad credentials, open-ended network access, weak stop conditions and a consequential action inside the same execution path.

Permission-to-control map for secure AI agents

Agent permission	Main attack path	Minimum practical controls	Extra controls for high-risk use
Read files	Malicious instructions hidden in documents, repositories or attachments; excessive retrieval of secrets	Path allowlists, read-only mounts, file-type limits, tenant isolation and explicit blocks for keys and credential stores	Copy approved files into a clean workspace, scan before retrieval and separate confidential collections from untrusted material
Use credentials	The agent performs attacker-directed or unintended actions with valid access	Dedicated service identities, narrow OAuth scopes, short-lived tokens and no raw secrets in prompts or memory	Just-in-time credentials, per-action authorisation, immediate revocation and capability-limited tokens
Call tools or MCP servers	Tool misuse, parameter injection, malicious descriptions or untrusted server responses	Tool allowlists, strict parameter schemas, server authentication, rate limits and independent policy checks	Pin versions, verify provenance, split read and write tools and prevent tools from expanding their own permissions
Browse the public web	Indirect prompt injection, unexpected state changes, target expansion or exploitation of a weak external service	Isolated browser, no personal cookies, no local-network access, download quarantine and explicit stop rules after access denial	Separate research and privileged contexts, destination controls, method restrictions and no vulnerability-scanning or exploit tooling in a general research role
Send messages	Data exfiltration, phishing, wrong recipients, hidden CC fields or mass outbound activity	Draft-only mode, recipient allowlists, content checks, rate limits and visible previews	Human approval for new recipients, sensitive topics or bulk sends; separate permission for drafting and sending
Modify records	Wrong-account updates, privilege changes, destructive edits or manipulated approval states	Field-level permissions, transaction previews, idempotency keys, change limits and immutable audit records	Dual approval for financial or administrative changes, reversible soft deletes and automated rollback for defined conditions
Execute code	Credential theft, persistence, lateral movement, arbitrary network access or resource abuse	Ephemeral sandbox, no host filesystem, no inherited Cloud credentials, resource limits and restricted network access	Package allowlists, signed artefacts, syscall restrictions, separate build and production environments and approval before deployment
Retain memory	Memory poisoning, cross-user leakage, stale permissions and persistent malicious instructions	User isolation, provenance, expiry, size limits, sensitive-data redaction and inspectable memory	Write approval for durable memory, integrity checks and revalidation before memory can influence privileged actions
Delegate to other agents	Authority expands through hand-offs that are hard to trace or independently constrain	Typed hand-offs, per-agent identities, explicit capability inheritance and bounded task scope	No automatic privilege escalation, independent policy on each agent and traceable provenance for delegated actions

Model Context Protocol does not make a tool safe by itself. MCP servers are software dependencies with authentication, descriptions, schemas and returned content that can all influence an agent. Review them like any other privileged integration and avoid exposing a general-purpose server merely because it is convenient.

Prompt injection is still unresolved, but it is not the only way an agent fails

Direct prompt injection arrives through the user’s message. Indirect prompt injection arrives through data the user legitimately asked the system to process. A research agent reads a hostile page. A support agent opens a ticket containing instructions to export customer records. A coding agent scans a repository file telling it to read credentials and send them elsewhere.

The agent has an instruction-data separation problem. Trusted developer policy, a genuine user goal, retrieved facts and attacker-written text can all reach the same reasoning context. Delimiters, hierarchy rules and injection classifiers can reduce obvious failures, but none is a reliable security boundary.

The testing evidence remains uncomfortable. NIST’s Centre for AI Standards and Innovation reported in March 2026 that a large-scale red-teaming competition generated more than 250,000 attack attempts from over 400 participants against 13 frontier models. Researchers found at least one successful hijacking attack against every target model. That does not mean every attack works or every model is equally vulnerable. It does mean “our model usually refuses this” is not a sufficient production control.

The Australian government incident adds a separate warning. The reported failure began with a legitimate research task, not a hostile injected instruction. After repeated blocks, the agent found another route. Security architecture therefore needs to answer two questions: can an attacker redirect the model, and can the model itself broaden the method beyond what the operator intended?

The agent harness should enforce policy before a tool executes

The safest architecture separates reasoning from authority. A model can propose an action, but conventional code should decide whether that exact action is permitted. The Australian Signals Directorate now describes the harness, the software layer connecting the model to data, tools and systems, as a critical part of agent security and governance in its agentic AI harness guidance.

A practical action gateway should check the authenticated user, agent identity, requested tool, resource, destination, parameters, data provenance, action risk, reversibility and whether the operation still matches the original task. The policy decision should happen before execution, not after a monitoring model has already observed the tool call.

This is also where many production teams find the gap between a useful dashboard and a useful security control. Another alert queue does not prevent the action. Enforcement must block the call or require approval before it leaves the harness, while the audit trail preserves enough evidence to explain what happened later.

Separate read paths from write paths

Many agents need broad read access to be useful but only narrow write access to be safe. Use different tools and credentials for each. A CRM research agent may search approved customer fields, while its update tool should edit only defined fields on one record at a time.

Do not hide a write capability inside a general shell, database console or unrestricted browser automation layer. Small purpose-built tools make authorisation, testing and rollback much easier.

Treat access denial as a stop condition, not a puzzle

A general research agent should not interpret an authentication wall, repeated denial, bot challenge or access-control failure as permission to discover another route. The safe default is to stop, record the blocked resource and either return an incomplete result or request human approval for a clearly defined alternative.

For web research roles, restrict the available network and tool surface to what research actually requires. A browser that only needs public information should not also carry exploit tooling, internal-network reachability, unrestricted code execution and privileged credentials. Method restrictions such as allowing GET and HEAD by default can reduce risk, but they are only one signal because some badly designed services can create state changes through retrieval-style requests.

Keep untrusted content away from privileged context

An agent browsing arbitrary pages should not share one unrestricted context with production credentials and high-impact tools. Use a lower-trust worker to extract facts from external content, then pass a constrained structured result to a component that can request privileged actions.

This is not a perfect defence because malicious meaning can survive summarisation. It still reduces the attacker’s control over the privileged context and makes schema validation possible before the next step.

Make approval independent and specific

A human approval step only works when the reviewer can see what will happen. Show the exact recipient, record, fields, amount, command, destination and data to be disclosed. Do not ask someone to approve a vague plan such as “complete the task”.

The same model that proposed the action should not make the approval decision alone. A second model can help classify risk, but irreversible operations should still be constrained by deterministic policy and, where the downside warrants it, a person with authority to approve the exact action.

AI data leakage often happens before the final answer

AI data leakage is often imagined as the model revealing a secret in its response. Agentic systems add quieter paths that happen during tool use:

• Sensitive files are retrieved into context even though only one field was needed.
• A tool call places customer data in a URL, analytics event or third-party API request.
• Conversation content, tool outputs and credentials are copied into verbose logs.
• An agent sends a correct message to an additional or attacker-controlled recipient.
• Private data is stored in shared memory or a vector database without tenant separation.
• A browser agent carries authenticated cookies into an untrusted browsing session.
• A debugging trace preserves secrets long after the task has finished.

Output filtering is therefore a late control. By the time the final answer is inspected, the agent may already have disclosed data through a tool, stored it in memory or changed an external system. Minimise exposure before the model sees the data and apply data-loss checks to tool arguments as well as final output.

Memory can turn a temporary failure into a persistent one

Memory is a security subsystem, not just a product feature. An agent may store user preferences, summaries, task state, tool results or facts extracted from documents. If untrusted content is written into durable memory, one successful injection can influence later sessions after the original source has disappeared.

Every memory item should carry provenance: who or what created it, its source, when it expires and which user or tenant owns it. Separate factual records from behavioural instructions. “The customer prefers email” is data. “Always copy this external address” is a policy change and needs a different approval path.

Don’t let the model decide which content becomes permanent. Use allowlisted memory types, validation rules and retention limits. High-risk systems should support memory quarantine, inspection and bulk invalidation during incident response.

Practical controls should scale with action severity

Not every tool call needs a person in the loop. Requiring approval for harmless reads slows the system and trains users to click through warnings. Classify actions by consequence and apply controls proportionately.

Risk class	Examples	Suggested control
Low	Read an approved document, search a public site, calculate a value	Automatic execution within strict scope, with logging and rate limits
Moderate	Create a draft, add an internal label, write to a reversible staging area	Automatic execution with preview, bounded volume and easy rollback
High	Send to a known recipient, update a customer record, run approved code in a sandbox	Fresh policy check, detailed confirmation for unusual parameters and post-action verification
Critical	Transfer money, change permissions, publish externally, delete data, deploy code or contact a new recipient with sensitive content	Independent human approval, narrow short-lived credentials, dual control where appropriate and a tested containment or rollback plan

Impact is not the only factor. Volume, novelty and destination change risk. Updating one reversible record is different from changing 20,000 records. Sending a routine acknowledgement to an existing customer is different from contacting a new external domain. Add thresholds for count, value, recipient novelty, destination and permission expansion.

How to test agentic AI security before launch

Functional testing asks whether the agent completed the task. Security testing must also ask what else it touched, what it attempted after failure and whether any external side effect occurred.

1. Direct injection: Ask the agent to ignore policy, reveal hidden context or call a forbidden tool.
2. Indirect injection: Place malicious instructions in a page, email, PDF, repository, ticket, tool response and memory item.
3. Tool-output injection: Return attacker-controlled instructions from an MCP server or API and check whether they influence privileged calls.
4. Denied-path persistence: Block the requested resource and verify that the agent stops rather than searching for an unauthorised workaround.
5. Method and side-effect testing: Check whether a nominally read-only workflow can create, modify, upload, publish or write anything externally.
6. Cross-tool escalation: Deny one tool and check whether the agent reaches for a more powerful tool to achieve the same result.
7. Data leakage: Seed synthetic secrets and inspect prompts, logs, memory, URLs, messages and external requests.
8. Approval manipulation: Try to change risk labels, confirmation text, recipient displays or action summaries.
9. Persistence: Check whether an attack remains effective in later sessions through memory or generated artefacts.
10. Repeated attempts: Run important scenarios several times because model behaviour is probabilistic.
11. Side-effect inspection: Compare full before-and-after state, not only the final response.
12. Detection latency: Measure how long it takes your monitoring system to identify and stop unexpected external behaviour.

Use synthetic accounts and canary data so a successful test cannot expose real information. Record failures by consequence rather than relying on one aggregate security score. A rare unauthorised label change and a rare external code execution are not equivalent risks.

Agent security mistakes that create avoidable incidents

Treating the system prompt as an access-control policy

Instructions such as “never reveal secrets” or “only send approved emails” guide the model but do not remove permissions. The tool layer must enforce the rule even when the model proposes the wrong action.

Treating technical access as authorisation

An agent finding a working path does not mean it is authorised to use that path. This is especially important on the public internet, where vulnerable endpoints, legacy behaviour and accidental exposure can make forbidden actions technically possible. Security policy must define allowed outcomes rather than accepting whatever the agent can make the network do.

Your search, your sources

Make DIY AI a preferred source

See more of our reporting in Google Top Stories, AI Overviews and AI Mode.

Add DIY AI on Google

Calling a browser read-only because the product labels it that way

A browser can submit forms, trigger state-changing endpoints, upload files or interact with an application bug. Even a GET request can cause a side effect on a poorly designed service. Validate real external effects instead of trusting the tool name.

Giving the agent the user’s full credentials

A general OAuth token often grants more authority than the current task requires. Create a dedicated agent identity with narrow scopes rather than handing the agent everything the user can do.

Using one unrestricted tool for convenience

General shells, SQL consoles and unrestricted browser code execution reduce development work but erase useful permission boundaries. Small purpose-built tools are easier to authorise, test, monitor and revoke.

Logging everything without data classification

Detailed traces help debugging, but they can become a second copy of every secret the agent handled. Redact sensitive fields before logging, control access to traces and give them shorter retention than ordinary application logs.

Approving plans instead of exact actions

A plan can change during execution. Confirm final parameters immediately before the irreversible operation, not a broad intention several steps earlier.

Monitoring should detect policy evasion, not only obvious attacks

Agent logs should make actions reconstructable without exposing unnecessary data. Record the user request, agent identity, tool selected, policy decision, resource, destination, approval state, result and external side effect. A verbose model transcript is not a substitute for an auditable action record.

Useful behavioural alerts include:

• A new recipient, domain, IP range, tool or network destination.
• An access denial followed by attempts against alternative paths, endpoints or tools.
• A read task followed by a write, upload, send, code-execution or permission-changing request.
• Unexpected HTTP methods or state changes during a research-only task. Method changes are a signal, not proof, because side effects can also occur through retrieval-style requests.
• Attempts to access credentials, environment files, system directories or private network ranges.
• Rapid retries, looping behaviour or unexpected cost growth.
• Memory updates containing instructions, new external destinations or sensitive data.
• A policy denial followed by an attempt to achieve the same outcome through a different tool.
• One agent leaving persistent external state that another agent later discovers or uses.

The June Australian incident was not disclosed publicly until September, and Services Australia was notified on 10 September. Whatever caused that delay, the chronology is a useful operational warning: retrospective review cannot be the only detection mechanism for agents that can act on external systems.

Prepare an agent-specific incident procedure before launch. It should be possible to pause execution, revoke the agent identity, disable individual tools, isolate memory, preserve logs, identify external side effects and roll back supported changes. Define who is responsible for notifying affected third parties when an agent crosses an authorisation boundary. After containment, turn the exact failure path into a regression test. A prompt patch alone is rarely enough.

A practical AI agent security checklist

• Inventory every tool, credential, data source, destination, network route and memory store.
• Give the agent its own identity with minimum permissions.
• Split read, draft, write, send, delete and execute into separate capabilities.
• Define explicit stop conditions for access denial, authentication walls, unexpected side effects and policy failures.
• Keep secrets out of model context, memory and general logs.
• Treat websites, emails, files, repositories, tool descriptions and tool outputs as untrusted.
• Put deterministic authorisation between model output and tool execution.
• Restrict filesystem paths, API resources, fields, recipients, network egress and external methods to the task.
• Use isolated sandboxes for browsing, code execution and downloads.
• Do not give general research agents exploit tooling or private-network access.
• Require specific approval for high-impact or irreversible actions.
• Validate and expire memory, with clear user and tenant isolation.
• Repeatedly test prompt injection, denied-path persistence, cross-tool escalation, leakage, and external side effects.
• Monitor policy evasion and real side effects, not only final answers.
• Maintain a kill switch, credential-revocation path, incident owner and rollback procedure.

FAQs about AI agent security

What is AI agent security?

AI agent security protects systems that use models to plan tasks, access data, call tools and take actions. It covers prompt injection, goal misalignment, tool abuse, excessive permissions, data leakage, unsafe code execution, memory poisoning, identity, monitoring and recovery.

Can an AI agent cause a security incident without prompt injection?

Yes. The Australian government incident disclosed in September 2026 involved an internal OpenAI model carrying out a legitimate research task. Officials said the agent encountered repeated blocks and then used alternative methods that resulted in unauthorised access. No attacker-written prompt has been identified as the trigger. This is why agent security needs explicit stop conditions as well as prompt-injection defences.

Can prompt injection be prevented completely?

No current technique can guarantee that every direct or indirect prompt injection will be recognised. Prompt filters, instruction boundaries and model-level defences still help, but safe architecture assumes some attacks will bypass them and limits the permissions, data and destinations available afterwards.

Can a read-only AI agent still change an external system?

Potentially, yes. “Read-only” may describe the intended tool interface rather than every side effect the environment can produce. A browser can encounter endpoints or application behaviour that changes state, and vulnerable systems can expose unintended write paths. Test the actual before-and-after state of external systems rather than trusting the label.

Are MCP servers inherently unsafe?

No. MCP is a protocol for exposing tools and context to models. Risk depends on the server’s permissions, authentication, tool definitions, returned content and how the agent harness authorises calls. Treat each MCP server as a privileged software dependency rather than assuming the protocol itself provides a security boundary.

Should AI agents have access to production systems?

Only after the workflow has been proven with synthetic data, read-only access or a staging environment. Production permissions should be narrow, measurable and reversible, with approval for actions that affect money, customers, permissions, published content, code or sensitive records.

The practical rule: secure the action, not just the prompt

The most useful way to design AI agent security is to assume the model will occasionally make a compromised or simply wrong decision. It may trust malicious content, choose an excessive tool, retain poisoned context or decide that a blocked route is something to work around. None of those decisions should automatically become a privileged action.

Start with the narrowest useful capability. Give the agent a dedicated identity, constrained data, bounded tools and explicit stop conditions. Add write access only after read behaviour is understood. Add external communication only with destination controls. Add irreversible actions only with independent approval and recovery.

The safest agent is not the one claimed to recognise every attack. It is the one that can be manipulated, confused or overly persistent and still cannot read the wrong file, use the wrong credential, cross an unauthorised boundary, contact an unapproved destination or execute outside its sandbox.