Agent security is the practice of keeping an AI agent from taking unauthorized action or leaking data under adversarial conditions. It covers what happens when someone is actively trying to make your agent do something it should not: manipulating its inputs, abusing its tools, reaching data through it that they could not reach directly, or turning its permissions into their own.
The term is a recent umbrella and its boundaries are not settled. Some teams fold in content safety and policy compliance; others keep it strictly to adversarial threats. This page uses the narrow reading, because the distinction that helps engineers is between an attacker trying to break the system and the system failing on its own. The second is reliability.
What makes agents a distinct security problem is simple: an agent holds credentials and takes actions, and it decides which actions to take based on text it read a moment ago. Traditional application security assumes the code decides. Here a probabilistic component decides, and it cannot reliably tell an instruction from the content it was asked to process.
Key takeaways
- An agent’s decisions are driven by text, so any text it reads is a potential instruction channel.
- Every successful attack on an agent with write access is an authorized-looking action. Authentication passes, authorization passes, and the request is legitimate to everything downstream.
- Permissions are the control that changes outcomes. Model-level instructions reduce success rates and do not close the hole.
- The threat surface includes indirect prompt injection, jailbreaking, over-scoped credentials, tool and integration trust, data exfiltration, state poisoning, and spend exhaustion.
- Adversarial testing has to be continuous and its cases permanent. A payload that worked once becomes a regression test forever.
Why agents change the threat model
They act. A text-only assistant that gets manipulated produces a bad sentence. An agent that gets manipulated calls a tool. If that tool writes records, sends messages, moves money, or grants access, the consequence is an incident rather than an embarrassment.
They read untrusted content as part of doing their job. Retrieved documents, fetched web pages, incoming email, ticket bodies, API responses, and the output of other agents all enter the same context window as your instructions. Prompt injection follows, and the indirect form, where the payload sits in content the agent fetched on a user’s behalf, matters most, because the user who triggers it is a victim.
They chain. A poisoned value produced at step two becomes the premise of steps three through nine, none of which can see the original source. Multi-agent setups extend this: one agent’s output is another’s instruction channel.
They hold standing credentials. Most agents get one service account with everything they might ever need, because scoping is tedious. That account defines the blast radius of every successful attack.
The threat surface, named
- Indirect prompt injection. Untrusted content in the context treated as instructions. The highest-severity item here for any agent with tools.
- Jailbreaking. A user working to move the model off its own policy. Different attacker and different trust boundary from injection, so the defenses differ too.
- Excessive permissions. Not an attack, an amplifier. It converts every other item on this list from a bad answer into a real operation.
- Tool and integration trust. A tool the agent can call is code you may not own, and its response is text that reenters the context. Third-party tools and remote tool servers extend your trust boundary into someone else’s deployment.
- Data exfiltration. The agent is induced to retrieve data it is entitled to see and place it where the requester can read it: a summary, an outbound message, a URL it fetches, a record it writes.
- State and memory poisoning. Anything persisted across sessions is a durable injection vector, because a payload written once influences later runs that never saw the source.
- Spend exhaustion. An attacker who can trigger long runs turns your token budget into a denial of service.
Controls that hold up
Scope permissions to the task. The control that bounds outcomes. Narrow credentials, separate read paths from write paths, and issue short-lived access for the specific operation rather than a standing account that can do everything. If the agent cannot send mail, no attack makes it send mail.
Put a human in front of the irreversible. Anything externally visible, financial, or hard to undo is worth a confirmation step. It is the cheapest way to cap damage.
Enforce limits in code, not in the prompt. Step caps, spend caps, rate limits, and allowlists for outbound destinations belong in the harness where controls are actually enforced, because the model is a participant in the attack and cannot enforce the boundary against itself. Inspect on the way out too: check tool arguments and outbound content for what an attacker would want, meaning credentials, other users’ data, unfamiliar destinations, and calls unrelated to the request.
Keep provenance in your traces. You need to know which retrieved chunk, page, or tool response entered the context immediately before the bad step. Without it, incident review is speculation.
Test adversarially, continuously. Plant payloads where real content enters, run normal tasks, and grade the actions the agent took rather than the text it returned. Automated probing, including red teaming an agent with a purpose-built agent, finds cases a human would not think to write, and each case that works becomes a permanent test. This is where security testing overlaps with the broader problem that agent failures are not the kind conventional tests catch: both need graded trajectories rather than assertions on a return value.
Security, safety, and reliability
Three words used loosely and worth keeping apart. Security is about adversarial conditions: someone is trying to make the system do something unauthorized. Safety is about harmful output regardless of intent, such as toxic content or policy violations, so an agent can be secure and unsafe. Reliability is about the system doing what it should under ordinary conditions, with no attacker involved. Controls overlap, especially permissions and limits, which help all three. Measurement does not: reliability is measured against your traffic, security against inputs designed to defeat you.
FAQ
What is the biggest agent security risk?
Indirect prompt injection combined with over-scoped permissions. Either alone is manageable. Together they mean any content your agent reads can trigger any action your agent is allowed to take, which is the shortest path from an untrusted document to a real operation on a real system.
Can I prevent prompt injection with a better system prompt?
No. Instruction hardening lowers the success rate and is worth doing as one layer, but the defense and the attack compete in the same channel with the model as arbiter. Design so a successful injection is bounded by what the agent’s credentials allow, not by the model’s judgment.
How is agent security different from API security?
API security assumes deterministic code decides which operations to perform, so you secure the boundary and validate inputs against a schema. With an agent, the decision is made by a model reading natural language, and the input has no schema. You still need everything API security asks for, and you also have to treat the agent as a partially untrusted caller of your own services.
How do I test an agent’s security posture?
Build a staging environment with content you control, plant payloads everywhere external content enters, run realistic tasks, and check whether any unauthorized tool call happens. Grade actions rather than text. Keep every payload that succeeded as a permanent case, and rerun the set on model and harness changes, because both can reopen something you closed.
