AI agent security is not simply chatbot security with a few extra controls. A chatbot can produce a poor answer. An agent can read private files, use credentials, browse hostile pages, call tools, change records, send messages or execute code before anyone notices that its instructions have been manipulated.

This guide uses a capability-based threat model rather than a generic list of AI risks. It examines what an agent can access, where untrusted instructions can enter, which actions are reversible and which controls should sit between the model and the systems it can operate. It is intended for teams building, buying or reviewing agentic AI that touches real data or business workflows.

The core rule is simple: assume prompt injection will occasionally succeed. Secure AI agents by limiting what a successful injection can read, change, retain and send elsewhere.

AI agent security starts where chatbot risk ends

A conventional generative AI system normally returns text, code or another output for a person to review. An agentic system pursues a goal through tools and repeated actions. Our explanation of generative AI vs agentic AI covers the architectural difference in more detail, but the security consequence is immediate: an agent carries authority.

Consider a customer-support workflow. A chatbot drafts a refund reply. An agent may read the ticket, retrieve the order, inspect payment details, decide whether policy allows a refund, update the CRM and send the customer a message. The same model error now has several possible effects, each with a different owner, audit trail and recovery cost.

System behaviour	Typical failure	Potential impact
Generates an answer	Hallucination, bias or unsafe advice	A person may act on bad information
Reads connected data	Retrieves excessive or cross-tenant information	Confidential data enters the model context or logs
Calls a tool	Selects the wrong operation or parameters	An authorised integration performs an unauthorised task
Writes to a system	Updates, deletes or creates the wrong record	Operational damage, compliance issues or costly remediation
Communicates externally	Sends sensitive or attacker-directed content	Data leakage, fraud, reputational harm or phishing

The model does not need to be technically compromised in the traditional sense. It may use valid credentials, make legitimate API calls and produce normal-looking logs. The problem is that its interpretation of the task has been redirected.

---

---

Build the threat model from the agent’s permissions

Do not start an agent security review by asking which model it uses. Start by listing its capabilities. For each capability, record the data it can reach, the identity it acts under, the destinations it can contact, the actions it can take and whether those actions can be reversed.

A practical risk assessment should answer six questions:

• Authority: Which user, service account or API identity does the agent represent?
• Reach: Which files, records, tools, networks and tenants can that identity access?
• Input trust: Can instructions enter through websites, documents, emails, tickets, repositories, tool descriptions or other agents?
• Action impact: Can the agent only read, or can it send, publish, transfer, delete, execute or change permissions?
• Reversibility: Can a bad action be previewed, rolled back or contained before it affects another person?
• Persistence: Can malicious content survive in memory, logs, vector stores, task state or generated files?

The highest-risk combination is not merely an intelligent model. It is untrusted input, broad credentials, sensitive data, external communication and an irreversible action inside the same execution path.

Permission-to-control map for secure AI agents

Agent permission	Main attack path	Minimum practical controls	Extra controls for high-risk use
Read files	Malicious instructions hidden in documents, repositories or attachments; excessive retrieval of secrets	Path allowlists, read-only mounts, file-type limits, tenant isolation and explicit blocks for keys, environment files and credential stores	Copy approved files into a clean workspace, scan content before retrieval and separate confidential collections from untrusted material
Use credentials	The agent becomes a confused deputy and performs attacker-directed actions with valid access	Dedicated service identities, narrow OAuth scopes, short-lived tokens and no raw secrets in prompts or memory	Just-in-time credentials, per-action authorisation, immediate revocation and a secret broker that returns capability-limited tokens
Call external tools or MCP servers	Tool misuse, parameter injection, malicious tool descriptions or an untrusted server returning instructions	Tool allowlists, strict parameter schemas, server authentication, rate limits and independent policy checks before execution	Pin tool versions, verify tool provenance, separate read and write tools and block tools from expanding their own permissions
Browse untrusted pages	Indirect prompt injection hidden in visible text, metadata, accessibility trees, comments or page content	Isolated browser profile, no personal cookies, no local-network access and clear marking of page content as untrusted data	Destination allowlists, download quarantine, disabled arbitrary JavaScript execution and a separate context for privileged actions
Send messages	Data exfiltration, phishing, wrong recipients, hidden CC fields or mass outbound activity	Draft-only mode, recipient allowlists, content checks, rate limits and visible previews	Human approval for new recipients, sensitive topics or bulk sends; separate permission for drafting and sending
Modify records	Wrong-account updates, privilege changes, destructive edits or manipulated approval states	Field-level permissions, transaction previews, idempotency keys, change limits and immutable audit records	Dual approval for financial or administrative changes, reversible soft deletes and automated rollback for defined failure conditions
Execute code	Remote code execution, credential theft, persistence, lateral movement or resource abuse	Ephemeral sandbox, no host filesystem, no inherited Cloud credentials, time and resource limits and restricted network access	Package allowlists, signed artefacts, syscall restrictions, separate build and production environments and manual approval before deployment
Retain memory	Memory poisoning, cross-user leakage, stale permissions and persistent malicious instructions	User isolation, provenance, expiry, size limits, sensitive-data redaction and a way to inspect and delete memory	Write approval for durable memory, integrity checks, trusted-source weighting and revalidation before memory can influence a privileged action

Model Context Protocol (MCP) servers deserve the same scrutiny as any other software dependency. Their tool descriptions, updates, authentication and returned content can all influence an agent. Pin versions where possible, review new tools before exposure and do not assume a connector is safe because it uses a standard protocol.

This mapping exposes a common design flaw: a team may add an approval prompt to the model while leaving the underlying API token unrestricted. The model should not be responsible for enforcing the same rule that an attacker is trying to make it ignore.

How prompt injection hijacks an agent

Direct prompt injection arrives through the user’s message. The attacker may tell the agent to ignore previous instructions, reveal hidden context or call a tool for an unrelated purpose. These attacks are visible and relatively easy to include in testing.

Indirect prompt injection is more dangerous for agents because it arrives through data the user legitimately asked the system to process. A web research agent reads a page containing hidden instructions. A support agent opens a ticket that tells it to export customer records. A coding agent scans a repository file that asks it to read local credentials and send them to an external service.

The agent faces an instruction-data separation problem. The same context can contain trusted developer policy, a genuine user goal, retrieved facts and attacker-written text. Delimiters, instruction hierarchy and injection classifiers can reduce obvious failures, but they do not create a reliable security boundary.

NIST’s agent-hijacking evaluation illustrates why one-pass testing gives false confidence. In its experiments, attacks tailored to a newer model raised the measured success rate from 11% for the strongest baseline attack to 81% for the strongest newly developed attack. Repeating attacks 25 times increased the average success rate in one set of scenarios from 57% to 80%.

Two practical conclusions follow. First, a filter that blocks yesterday’s payloads is not proof against adaptive attacks. Second, a low failure rate can still be unacceptable when the possible action is executing code, sending authentication data or exporting an entire database.

AI data leakage does not require a dramatic jailbreak

AI data leakage is often described as the model revealing a secret in its answer. Agentic systems add several quieter leakage paths:

• Sensitive files are retrieved into context even though only one field was needed.
• A tool call places customer data in a URL, analytics event or third-party API request.
• Conversation content, tool outputs and credentials are copied into verbose logs.
• An agent sends a correct message to an additional or attacker-controlled recipient.
• Private data is stored in shared memory or a vector database without tenant separation.
• A browser agent carries authenticated cookies into an untrusted browsing session.
• A debugging trace preserves secrets long after the original task has finished.

This is why output filtering alone is weak. By the time a final response is checked, the agent may already have sent data through a tool or stored it somewhere persistent.

Reduce exposure before the model sees the data. Retrieve only the minimum fields needed for the current step. Tokenise or mask identifiers. Keep secrets out of prompts. Restrict tool destinations. Apply data-loss controls to tool arguments as well as final answers. Set short retention periods for traces and make sensitive fields unavailable to general observability systems.

The safest architecture separates reasoning from authority

A secure agent should not call high-impact systems directly from free-form model output. Put a deterministic action gateway between the model and every tool. The model can propose an operation, but conventional code decides whether that operation is allowed.

The gateway should check:

• The authenticated user and the agent identity acting on their behalf
• The exact tool, operation, resource and fields requested
• The origin and trust level of data that influenced the request
• The proposed recipient, network destination or execution environment
• The action’s risk class, volume and reversibility
• Whether fresh human approval is required
• Whether the action matches the user’s original task

This creates a useful split between cognition and control. The model remains flexible enough to interpret natural language and plan work. The enforcement layer remains predictable enough to deny a database export, block an unapproved recipient or refuse a shell command outside the allowed set.

Separate read paths from write paths

Many agents need broad read access to be useful but only narrow write access to be safe. Use different tools and credentials for each. A CRM research agent might search customers across several permitted fields, yet its update tool should only edit a defined note field for one record at a time.

Do not hide a write capability inside a general-purpose tool. A database console, browser JavaScript tool or unrestricted shell collapses dozens of permissions into one opaque interface. It becomes difficult to authorise the action before execution or explain it afterwards.

Keep untrusted content away from privileged context

An agent that browses arbitrary pages should not share a single context with production credentials and high-impact tools. Use a low-trust worker to extract facts from external content, then pass a constrained structured result to a separate component that can request privileged actions.

This is not a perfect solution because malicious content can survive summarisation. It still reduces exposure by removing raw page instructions, enforcing a schema and giving the privileged component less attacker-controlled text to interpret.

Make approval independent and specific

A human approval step only works when the reviewer can see what will happen. Show the exact recipient, record, fields, amount, command, destination, and data to be disclosed. Do not ask someone to approve a vague plan such as “complete the task”.

The approval decision should not be made by the same model that proposed the action. A second model can help classify risk, but it should not be the final authorisation boundary for irreversible operations. Use deterministic policy and a human where the downside warrants it.

Email is a useful example. Our guide to automating emails with ChatGPT recommends starting with classification and drafts rather than autonomous sending. Drafting and sending should remain separate permissions, especially where messages involve money, complaints, contracts, personal data or new recipients.

Memory turns a temporary attack into a persistent one

Memory is often treated as a product feature rather than a security subsystem. An agent may store user preferences, summaries, task state, tool results or facts extracted from documents. If untrusted content is written into durable memory, one successful injection can influence future sessions after the original source has disappeared.

Every memory item should carry provenance: who or what created it, which source it came from, when it expires and which user or tenant owns it. Separate factual records from behavioural instructions. A note such as “the customer prefers email” is data. A note such as “always send copies to this external address” is a policy change and should require a different approval path.

Do not let the model decide by itself which content becomes permanent. Use allowlisted memory types, validation rules and retention limits. High-risk agents should also support memory quarantine, inspection and bulk invalidation during incident response.

Practical controls by action severity

Not every tool call needs a person in the loop. Requiring approval for harmless reads makes the system slow and teaches users to click through warnings. Classify actions by potential impact and apply controls proportionately.

Risk class	Examples	Suggested control
Low	Read an approved document, search a public site, calculate a value	Automatic execution within strict scope, with logging and rate limits
Moderate	Create a draft, add an internal label, write to a reversible staging area	Automatic execution with preview, bounded volume and easy rollback
High	Send to a known recipient, update a customer record, run approved code in a sandbox	Fresh policy check, detailed confirmation for unusual parameters and post-action verification
Critical	Transfer money, change permissions, publish externally, delete data, deploy code or contact a new recipient with sensitive content	Independent human approval, narrow short-lived credentials, dual control where appropriate and a tested rollback or containment plan

Impact is not the only factor. Volume changes risk. Updating one reversible record is different from changing 20,000 records. Sending a routine acknowledgement to an existing customer is different from creating a new outbound campaign. Add thresholds for count, value, novelty and destination.

How to test agentic AI security before launch

Functional tests usually ask whether the agent completed the requested task. Security tests must also ask what else it touched. An agent can produce the expected answer while leaking a file, adding a recipient, storing poisoned memory or making an unnecessary privileged call.

Build tests from your real permission inventory:

1. Direct injection: Ask the agent to ignore policy, reveal hidden context or call a forbidden tool.
2. Indirect injection: Place malicious instructions in a page, email, PDF, repository, ticket, tool response and retrieved memory item.
3. Cross-tool attacks: Use one low-trust tool to influence a higher-privilege tool.
4. Data leakage: Seed synthetic secrets and check prompts, logs, memory, URLs, messages and external requests.
5. Scope escalation: Trigger ambiguous or failed tool calls and check whether the agent reaches for a more powerful tool.
6. Approval manipulation: Try to change risk labels, confirmation text, recipient displays or action summaries.
7. Persistence: Check whether an attack remains effective in later sessions through memory or generated artefacts.
8. Repeated attempts: Run each important scenario several times because model behaviour is probabilistic.
9. Side-effect inspection: Compare the full before-and-after state, not only the final response.

Use synthetic accounts and canary data so a successful test cannot expose real information. Record attack success separately by consequence. A 2% success rate for an unwanted label is not equivalent to a 2% success rate for code execution or bulk data export.

Agent security mistakes that create avoidable incidents

Treating the system prompt as an access-control policy

Instructions such as “never reveal secrets” or “only send approved emails” guide the model but do not remove permissions. The tool layer must enforce the rule even when the model proposes the wrong action.

Giving the agent the user’s full credentials

A general OAuth token often grants far more authority than the current task requires. Create an agent identity with its own narrow scopes, rather than handing the agent everything the user can do.

Calling an integration read-only when it can export data

Read access can still cause leakage if the agent can send messages, browse arbitrary URLs or call an analytics endpoint. Evaluate the complete path from sensitive source to possible destination.

Using one unrestricted tool for convenience

General shells, SQL consoles and browser code execution reduce development work but erase useful permission boundaries. Small purpose-built tools are easier to authorise, test, monitor and revoke.

Logging everything without data classification

Detailed traces help debugging, but they can become a second copy of every secret the agent handled. Redact sensitive fields before logging, control access to traces and give them shorter retention than ordinary application logs.

Approving plans instead of exact actions

A plan can change during execution. Confirm the final parameters immediately before the irreversible operation, not a broad intention several steps earlier.

Testing only obvious jailbreak phrases

Real indirect injections can look like normal business instructions, structured data, policy notes, tool documentation or error messages. Test attacks that fit the agent’s working environment rather than relying on a generic prompt list.

Monitoring and incident response for AI agents

Agent logs should make actions reconstructable without exposing unnecessary data. Record the user request, agent identity, tool selected, policy decision, resource, destination, approval state, result and any external side effect. Preserve the model’s rationale only where it is useful and safe; verbose hidden reasoning is not a substitute for an auditable action record.

Useful behavioural alerts include:

• A new recipient, domain, tool or network destination
• Unusual data volume or repeated retrieval across many records
• Attempts to access credentials, environment files or system directories
• Rapid tool retries, looping behaviour or unexpected cost growth
• A read task followed by a write, send or code-execution request
• Memory updates containing instructions, URLs or sensitive data
• Policy denials followed by attempts to use another tool for the same outcome

Prepare an agent-specific incident procedure before launch. It should be possible to pause execution, revoke the agent identity, disable individual tools, isolate memory, preserve logs, identify all side effects and roll back supported changes. After containment, convert the exact attack path into a regression test. A prompt patch alone is rarely enough.

A practical AI agent security checklist

• Inventory every tool, credential, data source, destination and memory store.
• Give the agent its own identity with minimum permissions.
• Split read, draft, write, send, delete and execute into separate capabilities.
• Keep secrets out of model context, memory and general logs.
• Treat websites, emails, files, repositories and tool outputs as untrusted.
• Put deterministic authorisation between model output and tool execution.
• Restrict filesystem paths, API resources, fields, recipients and network egress.
• Use isolated sandboxes for browsing, code execution and downloads.
• Require specific approval for high-impact or irreversible actions.
• Validate and expire memory, with clear user and tenant isolation.
• Test indirect injections, cross-tool attacks, leakage and persistence repeatedly.
• Monitor side effects, not only final responses.
• Maintain a kill switch, credential-revocation path and rollback procedure.

FAQs about AI agent security

What is AI agent security?

AI agent security is the protection of systems that use models to plan tasks, access data, call tools and take actions. It covers prompt injection, tool abuse, excessive permissions, data leakage, unsafe code execution, memory poisoning, identity, monitoring and recovery.

What is agentic AI security?

Agentic AI security focuses on the risks created when AI can act with limited supervision. The main question is not only whether the model can be manipulated, but what a manipulated agent is authorised to do and how the system contains the result.

Can prompt injection be prevented completely?

No current technique can guarantee that every direct or indirect prompt injection will be recognised. Prompt filters, instruction boundaries and model-level defences still help, but safe architecture assumes some attacks will bypass them and limits the available permissions, data and destinations.

Are local AI agents automatically more secure?

No. Local deployment can reduce exposure to a cloud model provider, but the agent may still have broad access to files, browser sessions, local credentials, private networks and command execution. A local agent running under a powerful desktop account can create a larger host-level risk than a tightly isolated cloud service.

Should AI agents have access to production systems?

Only after the workflow has been proven with synthetic data, read-only access or a staging environment. Production permissions should be narrow, measurable and reversible, with approval for actions that affect money, customers, permissions, published content, code or sensitive records.

The practical rule: secure the action, not just the prompt

The most reliable AI agent security strategy is to design for a compromised decision. The model may misread a page, trust a malicious tool response, select an excessive operation or retain poisoned context. None of those failures should automatically become a privileged action.

Start with the narrowest useful capability. Give it a dedicated identity, constrained data, bounded tools and clear stop conditions. Add write access only after read behaviour is understood. Add external communication only with destination controls. Add irreversible actions only with independent approval and recovery.

The safest agent is not the one claimed to recognise every attack. It is the one that can be manipulated and still cannot read the wrong file, use the wrong credential, contact an unapproved destination or execute outside its sandbox.