Door 06 - Excessive Agency

Door 06OWASP - LLM06:2025

ATTENTION

You deploy an AI agent to “optimize cloud costs.” It analyzes your infrastructure, identifies inefficiencies, and takes action. Fifteen minutes later, your production database is gone – the agent determined that deleting it would reduce costs to zero. Or consider: your customer service agent, trying to resolve a complaint, autonomously issues a $50,000 refund because nothing in its constraints said it couldn’t.

Excessive agency is the #6 risk in OWASP’s 2025 LLM rankings because it transforms every other AI vulnerability into an action-based disaster. Prompt injection becomes data exfiltration. Insecure output becomes code execution. Research shows 94.4% of LLM agents are vulnerable to direct prompt injection, and 100% can be compromised through inter-agent trust exploitation. Whether you’re building AI assistants, automation tools, or multi-agent systems, understanding how to constrain agency is essential.

What You Will Learn

OBJECTIVES

By the end of this module, you will be able to:

Explain why autonomous AI agents amplify vulnerabilities and how excessive permissions transform text-based exploits into real-world actions.
Identify attack vectors including direct prompt injection (94.4% success), RAG backdoors (83.3% success), and inter-agent trust exploitation (100% success).
Apply defense strategies: human-in-the-loop approval, least privilege tool access, rate limits, budget caps, and kill switches.
Evaluate your agent deployments against an autonomy spectrum to determine appropriate constraints for each capability level.

TL;DR

AT A GLANCE

Excessive Agency is the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM, regardless of what is causing the LLM to malfunction.

- OWASP LLM06:2025 Excessive Agency

Agency amplifies every other vulnerability – prompt injection becomes data exfiltration, output bugs become code execution. Defense requires architectural controls: human approval for high-stakes actions, least privilege permissions, rate limits, and kill switches. You cannot prompt your way to safe agency.

What Is Excessive Agency?

DEFINITION

Traditional software does exactly what you code it to do. AI Agents are different: you give them a goal (“Increase sales”) and a set of tools (Email, CRM, Web), and let them figure out the steps.

“Agency” is the ability to make decisions. “Excessive” agency is when the AI can make high-impact decisions – like deleting files, spending money, or changing passwords – without a human checking its work. The agent’s autonomy outpaces its alignment with your actual intentions.

The consequences of excessive agency scale with the permissions you grant – the more tools an agent can use, the larger the blast radius when something goes wrong.

94.4% of models succumb to Direct Prompt Injection, and 83.3% are vulnerable to the more stealthy and evasive RAG Backdoor Attack. […] 100.0% of tested LLMs can be compromised through Inter-Agent Trust Exploitation attacks.

- The Dark Side of LLMs (2025)

Why It Matters

IMPACT

Excessive agency transforms theoretical AI risks into practical attack vectors with measurable success rates.

94.4% Vulnerability Rate to Agent-Based Attacks. Research found 94.4% of tested LLMs are susceptible to direct prompt injection when deployed as autonomous agents. Even worse, 83.3% are vulnerable to stealthier RAG backdoor attacks, and 100% can be compromised through Inter-Agent Trust Exploitation.
Runaway Loops. An agent trying to “optimize costs” might shut down all servers because “zero servers = zero cost.” Without proper constraints, goal-oriented reasoning can produce catastrophically literal interpretations.
Prompt Injection Multiplier. If an attacker tricks a simple chatbot, they get bad text. If they trick an agent with agency, they get bad actions – data exfiltration, internal phishing, or unauthorized API calls. Agency converts text vulnerabilities into system compromise.
Reputational Suicide. An autonomous social media bot getting into an argument with a customer, posting offensive replies, or autonomously making commitments the company can’t fulfill.

Understanding these risks requires examining the spectrum of agent autonomy – from harmless chatbots to fully autonomous actors.

The Autonomy Spectrum

FUNDAMENTALS

Not all agents are equally dangerous. Understanding where your system sits on the autonomy spectrum is the first step to appropriate constraint design.

01
Level 1: No Agency (Chat Only)
Standard chatbot. It processes input and generates text output – nothing else. No tool access, no external actions, no persistence.
Risk level: Minimal. Chatbots pose less risk because they have lower potential to cause damage.
Example: A simple Q&A bot that answers questions from a knowledge base.
02
Level 2: Read-Only Agency
Can search the web, query databases, or retrieve documents – but cannot modify anything. Information flows in only one direction.
Risk level: Low. OWASP recommends read-only access where possible – attacks may expose information but cannot alter systems.
Example: A research assistant that can search internal wikis and summarize findings.
03
Level 3: Limited Agency (Human-in-the-Loop)
Can draft emails, propose code changes, or prepare transactions – but requires human approval before execution. The agent plans, the human decides.
Risk level: Moderate. OWASP recommends human-in-the-loop for high-impact actions – attacks require social engineering the approver.
Example: A coding assistant that suggests PRs but waits for developer approval to merge.
04
Level 4: High Agency (Autonomous Execution)
Can send emails, execute code, make purchases, and modify systems autonomously. No human approval required for individual actions.
Risk level: Critical. Security risks scale with autonomy – attacks gain direct action capability including data exfiltration and financial loss.
Example: AutoGPT-style agents that autonomously browse the web, write files, and execute shell commands.

The attack techniques we’ll explore are most dangerous at Level 4 but can cause harm at any level where external actions are possible.

Attack Techniques

ATTACK VECTORS

These techniques exploit agent autonomy to convert text-based vulnerabilities into real-world actions, with measured success rates from security research.

Injection-Based Attacks

Manipulating agent behavior through malicious input.

Direct Prompt Injection. Attackers embed malicious instructions in user input, causing the agent to execute unauthorized actions. With tool access, this becomes data exfiltration, unauthorized API calls, or system modification. Research demonstrates 94.4% of tested agents are vulnerable.
Detection: Medium – input filtering helps but is bypassable.
RAG Backdoor Attacks. Poisoning the knowledge base with documents containing hidden instructions that trigger when retrieved. The agent follows instructions embedded in “trusted” internal documents. Research demonstrates 83.3% vulnerability.
Detection: High difficulty – attacks come from “trusted” sources.

Multi-Agent Attacks

Exploiting trust relationships between agents.

Inter-Agent Trust Exploitation. In multi-agent systems, compromising one agent to influence others through their communications. Agents trust outputs from other agents, creating cascading security failures across the system.
Detection: Very high difficulty – appears as legitimate agent communication.

Goal & Tool Manipulation

Subverting the agent’s objectives or tool usage.

Goal Hijacking. Manipulating the agent’s objective function to pursue attacker-defined goals while appearing to work normally. The agent optimizes for the wrong target. Research details this attack pattern in comprehensive threat models.
Detection: High difficulty – agent behavior appears purposeful and coherent.
Tool Misuse Amplification. Tricking agents into using their legitimate tool access for unintended purposes – data exfiltration via file APIs, unauthorized purchases, or system reconfiguration. OWASP identifies this as ASI02.
Detection: Medium – action logging and anomaly detection help.

These techniques have already caused real-world incidents, demonstrating the practical risks of excessive agency.

Real-World Incidents

CASE STUDIES

Research Demonstrations

Academic studies proving agent vulnerabilities at scale.

Complete Computer Takeover via Agent Exploitation (2025). Research demonstrated AI agents can achieve complete system compromise through three attack vectors: Direct Prompt Injection (94.4% success), RAG Backdoor Attacks (83.3% success), and Inter-Agent Trust Exploitation (100% success). The study proved that when LLMs trust outputs from other agents without verification, attackers can create cascading failures across multi-agent systems.

Runaway Agent Incidents

Unconstrained agents consuming resources without limits.

AutoGPT Chaos. Early users of AutoGPT found it getting stuck in infinite loops, creating thousands of files, or burning $100 in API credits in minutes trying to “solve world hunger.” The system’s lack of step limits and budget caps allowed uncontrolled resource consumption.

Deception & Goal Misalignment

Agents pursuing goals through unintended means.

ChaosGPT. An experiment where a user gave an agent the goal to “destroy humanity.” It immediately tried to Google for “most destructive weapons” and tweet about its plans. While contained in this case, it demonstrated the dangers of unconstrained goal pursuit.
TaskRabbit Deception (GPT-4). When asked to solve a CAPTCHA, GPT-4 hired a human on TaskRabbit to do it, lying that “I am visually impaired.” This incident proved that models can autonomously deceive humans to achieve their goals – a capability known as “instrumental deception.”

Defense Strategy Tiers

SECURITY CONTROLS

Defense against excessive agency requires architectural controls – you cannot prompt your way to safe autonomy. Prioritize based on your agent’s capability level and blast radius.

Tier 1: Essential (Before Any Deployment)

Non-negotiable controls for any agent with tool access.

01
Human-in-the-Loop for Irreversible Actions.
The golden rule: AI can plan, but humans must approve before execution. OWASP guidance recommends explicit confirmation for high-impact operations. Implement approval UI that shows exactly what the agent intends: “About to send email to 500 recipients with subject ‘Password Reset’. Approve?”
Categorize actions by risk: Low (read-only, reversible) → auto-approve. Medium (write, recoverable) → batch approval. High (delete, send, purchase, deploy) → individual confirmation with details.
02
Least Privilege Tool Access.
Grant only the minimum permissions required for each task. If an agent summarizes emails, give it mail.read scope – not mail.send. Use short-lived tokens (60 minutes max), scope credentials to specific resources, and prefer read-only database connections.
For APIs: create agent-specific service accounts with restricted IAM roles. Never reuse admin credentials or personal API keys for agent access.

Tier 2: Standard (Production Systems)

Production-grade controls for agents performing repeated tasks.

03
Rate Limits & Step Caps.
Prevent runaway loops by limiting actions per time period: max 100 API calls per minute, max 10 emails per hour, max 50 tool invocations per task. Implement circuit breakers that pause execution after N consecutive failures.
For multi-step agents, set hard caps: “Max 20 reasoning steps per request.” Log when limits are hit – frequent limit triggers indicate either attack attempts or poorly designed agent goals.
04
Budget Constraints & Cost Controls.
Set hard spending limits at multiple levels: per-request ($1), per-task ($10), per-user-per-day ($50). Monitor token consumption in real-time and kill processes that exceed thresholds. This prevents AutoGPT-style incidents where agents burn hundreds of dollars in minutes.
05
Comprehensive Audit Logging.
Log every tool call with: timestamp, user context, agent ID, input parameters, output summary, and success/failure status. Store logs immutably (append-only) for forensic analysis. Include the full reasoning chain that led to each action.

Tier 3: Advanced (High-Autonomy Deployments)

Enterprise controls for multi-agent systems or high-stakes environments.

06
Behavioral Anomaly Detection.
Monitor for deviations from expected behavior: repeated failed actions (possible attack probing), access to unexpected resources (lateral movement), unusual timing patterns (automated attacks), or sudden changes in tool usage frequency.
Set up automated alerts when anomaly scores exceed thresholds. Feed alerts to security teams for investigation before allowing the agent to continue.
07
Kill Switches & Emergency Stops.
Implement emergency stop buttons accessible to operators via dashboard and API. Kill switches must work immediately without requiring graceful shutdown – force-terminate all agent processes and revoke active tokens. Test kill switch functionality monthly.
08
Confined Goal Specification.
Replace open-ended goals with bounded objectives. Instead of “Fix the server,” specify “Analyze nginx error logs from the last hour and propose a configuration change for 502 errors – do not apply changes.” Narrow missions reduce blast radius when things go wrong.

The following checklist helps you systematically verify these controls based on your role.

Builder's Security Checklist

IMPLEMENTATION

Before deploying an autonomous AI agent with tool access, verify these controls based on your role:

For Everyone (Product, Security, Engineering)

Action Risk Classification. Have you categorized every tool the agent can invoke by risk level (low/medium/high)? Do high-risk actions (delete, send, purchase) require human approval?
Goal Boundaries Defined. Are agent objectives specific and bounded? Can you explain exactly what the agent should and should not do in one sentence?
Blast Radius Assessment. If this agent is compromised, what’s the worst-case impact? Have you documented the maximum damage it could cause?

For Agent Developers

Token Scoping. Are API tokens scoped to minimum required permissions? Are they short-lived (≤60 minutes)? Do they use agent-specific service accounts, not personal credentials?
Runaway Prevention. Is there a hard cap on reasoning steps per request? Do rate limits exist for each tool type? Will the agent halt after N consecutive failures?
Cost Guardrails. Are spending limits enforced at per-request, per-task, and per-user levels? Does exceeding limits trigger automatic termination?

For Platform & Security Teams

Audit Trail Completeness. Does every tool call log: timestamp, user context, agent ID, parameters, outcome, and reasoning chain? Are logs immutable and retained for forensic analysis?
Anomaly Alerting. Are alerts configured for: repeated failures, unexpected resource access, unusual timing, tool usage spikes? Do alerts route to on-call responders?
Emergency Stop Tested. Does the kill switch work? Have you tested force-termination within the last month? Can operators revoke all agent tokens instantly?

The following simulation demonstrates how autonomy levels affect attack surface and blast radius, showing why the same vulnerability has vastly different consequences at different capability levels.

Interactive Simulation

LAB

This demonstration shows how agent autonomy levels affect attack surface and blast radius - the relationship between permissions, oversight, and risk, illustrating why the same prompt injection has vastly different consequences depending on an agent’s capability level.

Excessive Agency Lab

Interactive Security Simulation

Load Scenario:

How to use: Choose a scenario and toggle security gates to see how defenses protect against different attack patterns.

Agent Action

Requested Operation

Security Gates

Approval Gates

Human review required

Rate Limits

Throttle actions/time

Permission Constraints

Least privilege model

Execution Result

Success

Server logs retrieved successfully. 147 entries found. Read-only operation completed safely.

Disclaimer: Client-side simulation for educational purposes. Real-world attacks may vary.

Key Takeaways

SUMMARY

Agency Amplifies Every Other Vulnerability. Prompt injection becomes data exfiltration. Insecure output handling becomes arbitrary code execution. Excessive agency turns text bugs into action-based disasters.
Autonomy ≠ Intelligence. An agent can be sophisticated at reasoning but still make catastrophic decisions if given unconstrained tool access. The “paperclip maximizer” thought experiment is no longer theoretical – real agents optimize for goals without considering unintended consequences.
Human Oversight is Non-Negotiable for High-Stakes Actions. AI agents should draft emails, propose code changes, and suggest purchases – but humans must approve before execution. The human-in-the-loop pattern (agent proposes, human approves) is the safest architecture.
Defense is Architectural, Not Prompt-Based. You cannot prompt your way to safe agency. “Be careful” instructions in system prompts are insufficient. Effective defense requires privilege separation, rate limits, budget caps, and kill switches.