Listen to this Post
Introduction:
The rapid adoption of agentic AI systems, where Large Language Models (LLMs) autonomously plan and execute complex tasks, introduces a new frontier of cybersecurity risks. This shift from passive chatbots to active, goal-oriented agents necessitates a fundamental redesign of security architectures to mitigate potential vulnerabilities in planning, tool execution, and external interactions. The OWASP Agentic Security Initiative provides critical frameworks for building these systems with resilience at their core.
Learning Objectives:
- Understand the core security principles behind the Plan-then-Execute architecture for LLM agents.
- Learn to implement secure command validation and sandboxing for agent-executed tools.
- Develop strategies to mitigate prompt injection, privilege escalation, and unauthorized tool use within AI agent workflows.
You Should Know:
1. Sandboxing Agent Tool Execution
A primary security risk is an agent being tricked into executing a malicious command on the host system. Always sandbox tool execution.
Code Snippet (Python using Docker):
import docker
def run_sandboxed_command(command):
client = docker.from_env()
container = client.containers.run(
"python:3.9-slim",
f"sh -c '{command}'",
detach=True,
remove=True, Auto-remove container after run
network_mode="none", Disable network access
mem_limit="100m", Limit memory
cpu_quota=10000 Limit CPU
)
logs = container.wait()
return container.logs()
Step-by-step guide:
This function uses the Docker Python SDK to create a disposable container with strict resource constraints and no network access. The agent’s intended command (e.g., curl example.com) is executed inside this isolated environment. The `network_mode=”none”` flag is critical to prevent the container from making any external network calls, neutralizing potential data exfiltration or reverse shell attacks. Always validate and sanitize the `command` input before passing it to this function to prevent container escape attempts.
2. Validating and Sanitizing LLM Planner Output
Before execution, the plan generated by the LLM must be rigorously validated against an allowlist of permitted tools and arguments.
Code Snippet (Python):
ALLOWED_TOOLS = {
"google_search": {"query": str},
"calculate_math": {"expression": str},
"read_file": {"filepath": str}
}
def validate_plan(plan_step):
tool_name = plan_step.get("tool")
if tool_name not in ALLOWED_TOOLS:
raise ValueError(f"Tool {tool_name} is not permitted.")
allowed_args = ALLOWED_TOOLS[bash]
for arg_key, arg_value in plan_step.get("args", {}).items():
if arg_key not in allowed_args:
raise ValueError(f"Argument {arg_key} is not permitted for {tool_name}.")
Type checking
if not isinstance(arg_value, allowed_args[bash]):
raise ValueError(f"Argument {arg_key} must be of type {allowed_args[bash]}.")
return True
Step-by-step guide:
This code defines a dictionary, ALLOWED_TOOLS, which acts as a strict policy allowlist. The `validate_plan` function checks each step in the agent’s proposed plan against this policy. It ensures the tool name exists in the allowlist and that each provided argument is both permitted and of the correct data type. This prevents an exploited LLM from attempting to call dangerous tools like `os.system` or passing malicious arguments to benign tools.
3. Securing API Keys for External Tool Calls
Agents often need API keys for external services. These must never be exposed to the LLM itself. Use environment variables and a secure proxy layer.
Bash Command to Set Environment Variables:
Set credentials securely in the environment export SEARCH_API_KEY="your_api_key_here" export WEATHER_API_KEY="another_key_here" Never hardcode keys in your application code!
Code Snippet (Python – Secure API Call):
import os
import requests
def secure_api_call(service, payload):
api_keys = {
"search": os.environ.get('SEARCH_API_KEY'),
"weather": os.environ.get('WEATHER_API_KEY')
}
key = api_keys.get(service)
if not key:
raise ValueError("Invalid service or API key not set.")
headers = {'Authorization': f'Bearer {key}'}
Make the call from the backend, the LLM only sees the response.
response = requests.post(f"https://api.{service}.com/v1/", json=payload, headers=headers)
return response.json()
Step-by-step guide:
API keys are loaded from the environment variables of the secure backend server, not from the application code or the LLM’s context. The agent’s plan might request a “google_search,” but the actual function call (secure_api_call) is handled by the backend, which injects the credential. The LLM only receives the sanitized result of the API call, ensuring credentials are never leaked in the prompt or by the model’s output.
4. Implementing Rate Limiting and Cost Controls
A malicious prompt could cause an agent to run in an infinite loop, generating enormous API costs. Implement circuit breakers.
Code Snippet (Python):
from functools import wraps
class BudgetTracker:
def <strong>init</strong>(self, max_cost=10.0):
self.cost = 0.0
self.max_cost = max_cost
def check_budget(self, estimated_cost=0.0):
if self.cost + estimated_cost > self.max_cost:
raise RuntimeError("Agent has exceeded its allocated budget.")
self.cost += estimated_cost
def budget_guard(estimated_cost=0.0):
def decorator(func):
@wraps(func)
def wrapper(args, kwargs):
budget_tracker = kwargs.get('budget_tracker')
if budget_tracker:
budget_tracker.check_budget(estimated_cost)
return func(args, kwargs)
return wrapper
return decorator
Decorate tool functions
@budget_guard(estimated_cost=0.1)
def call_expensive_api(query):
... function logic ...
return result
Step-by-step guide:
The `BudgetTracker` class monitors the total cost of an agent’s execution. The `budget_guard` decorator is applied to every function that incurs a cost (e.g., API calls, LLM inferences). Before the function executes, it checks with the budget tracker. If the action would exceed the predefined maximum budget, the function raises an exception and the agent’s execution is halted. This prevents financial damage from a prompt injection attack designed to waste resources.
5. Mitigating Prompt Injection in the Agent Loop
A persistent threat is indirect prompt injection, where malicious data from a external source (e.g., a webpage the agent reads) hijacks the agent’s subsequent steps.
Code Snippet (Python – Context Separation):
def execute_agent_loop(user_input):
Step 1: Planning - LLM generates a plan based on SANITIZED user input only.
plan_prompt = f"""
User Request: {user_input}
Generate a plan using only allowed tools.
"""
plan = llm.generate(plan_prompt)
Step 2: Sanitize all external data before feeding it back to the LLM.
for step in plan:
tool_result = execute_tool(step)
Critical: Treat all external data as untrusted
sanitized_result = sanitize_input(tool_result)
Step 3: Only the sanitized result is added to the context for the next LLM call.
plan_prompt += f"\nTool Result: {sanitized_result}"
Final execution and response
return llm.generate(plan_prompt)
Step-by-step guide:
This simplified loop demonstrates the principle of context separation. The initial user input is considered the only trusted directive. All data gathered from external tools during execution (e.g., search results, file contents) is treated as untrusted and potentially hostile. Before this data is fed back to the LLM for the next step, it must be sanitized (sanitize_input), which could involve truncation, removing HTML/XML tags, or using a separate classifier to detect injection attempts. This breaks the chain of trust that allows indirect injections to propagate.
What Undercode Say:
- Security is a Architectural Foundation, Not a Feature. The most critical takeaway is that security cannot be bolted onto an agentic system after development. The Plan-then-Execute pattern is inherently more secure than ReAct-style architectures because it creates a natural choke point for validation and approval before any action is taken. This design forces a separation of concerns between the “thinking” (planning) and “doing” (execution) phases.
- Assume the LLM is Compromised. The entire security model must operate on the assumption that the LLM planner can and will be tricked into generating malicious plans. Therefore, the trust is placed not in the LLM’s output, but in the validation layer that scrutinizes every command, the sandbox that contains every execution, and the policies that strictly enforce least privilege. This zero-trust approach applied to the AI itself is the cornerstone of resilient design.
Analysis: The move towards agentic AI represents a paradigm shift in application security. Traditional vulnerabilities like SQL injection have analogs in tool injection, but the attack surface is broader and more dynamic. The guidance from OWASP provides a crucial framework for this new era. By implementing strict input validation, robust sandboxing, and principled architectural patterns like Plan-then-Execute, developers can harness the power of autonomous agents while maintaining critical security controls. The techniques outlined here, from budget tracking to context sanitization, are essential tools for building the next generation of secure AI applications.
Prediction:
The sophistication of prompt injection and agent hijacking attacks will grow exponentially, mirroring the evolution of traditional malware. We will see the emergence of AI-specific worms capable of propagating through multi-agent systems by using one compromised agent to inject malicious prompts into others. This will necessitate the development of advanced AI-native security tooling, such as real-time plan validators using smaller, specialized models and automated red teaming systems that continuously stress-test agentic loops for unforeseen vulnerabilities, making security an integral and continuous part of the AI development lifecycle.
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Ronaldfloresdelrosario Owasp – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
