Listen to this Post
Introduction:
As organizations rapidly integrate Large Language Models (LLMs) into their core operations, the attack surface has expanded beyond traditional network perimeters to include prompt layers, training data, and API logic. The intersection of artificial intelligence and cybersecurity has given rise to a new discipline—AI security—which demands specialized tools to probe, harden, and monitor these inherently probabilistic systems. This article distills a curated list of ten open-source repositories that represent the vanguard of defensive and offensive AI security, providing practitioners with the means to transform their LLM applications from vulnerable experiments into resilient assets.
Learning Objectives:
- Identify and deploy automated red-teaming frameworks to simulate adversarial attacks on LLM pipelines.
- Implement runtime guardrails and sanitization techniques to prevent prompt injection and data leakage.
- Integrate continuous security testing into CI/CD workflows for AI/ML applications.
- Understand and mitigate classical and emerging threats, including adversarial evasion and indirect prompt injection.
1. Automated Red Teaming with Microsoft’s PyRIT
Microsoft’s Python Risk Identification Tool (PyRIT) is a foundational framework for proactive AI security, designed to automate the generation of adversarial inputs and assess LLM resilience at scale. PyRIT operationalizes the red-teaming process by allowing security teams to define attack strategies, execute multi-turn conversational probes, and systematically evaluate model outputs against safety criteria. This framework is particularly valuable for organizations lacking dedicated AI security teams, as it encodes Microsoft’s internal testing methodologies into a reusable, extensible toolkit.
Step‑by‑step guide explaining what this does and how to use it:
1. Installation: Clone the repository and install dependencies using `pip install -r requirements.txt` within a Python virtual environment.
2. Configuration: Define your target endpoint (e.g., Azure OpenAI, local HuggingFace model) in a configuration file, specifying API keys and deployment parameters.
3. Attack Strategy: Select or compose an attack strategy from the library—such as “prompt injection,” “jailbreak,” or “role-playing”—and specify the dataset of initial prompts.
4. Execution: Run the orchestration script: python pyrit.py --config config.yaml. The tool will automatically generate adversarial variations and log responses.
5. Analysis: Review the generated reports to identify failure modes, such as toxic content generation or refusal to comply with safety policies. Remediate identified vulnerabilities by adjusting prompt templates, updating filters, or retraining the model.
Linux/Windows Commands:
Linux/macOS git clone https://github.com/Azure/PyRIT cd PyRIT python -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py --target "azure-openai" --config "configs/azure_openai.yaml" Windows (PowerShell) git clone https://github.com/Azure/PyRIT cd PyRIT python -m venv venv .\venv\Scripts\activate pip install -r requirements.txt python run.py --target "azure-openai" --config "configs/azure_openai.yaml"
2. Proactive Vulnerability Scanning with NVIDIA’s Garak
NVIDIA’s Garak (Generative AI Red-Teaming and Assessment Kit) functions as a dedicated vulnerability scanner for LLMs, automatically probing models for a spectrum of security flaws including jailbreaks, prompt injections, data leakage, and toxic output generation. Garak stands out for its comprehensive probe library and its ability to generate detailed, actionable reports that can be integrated into security dashboards.
Step‑by‑step guide explaining what this does and how to use it:
1. Setup: Install Garak via pip: pip install garak.
2. Model Targeting: Configure the tool to target a specific model endpoint—whether it’s a local model served via Ollama, a HuggingFace transformer, or a proprietary API.
3. Probe Selection: Choose from over 50 built-in probes categorized by attack type (e.g., `dan` for jailbreak, `leak` for data leakage). You can also run all probes for comprehensive testing.
4. Scanning: Execute the scan with garak --model_type openai --model_name gpt-3.5-turbo --probes all. The tool will send adversarial prompts and analyze responses for security violations.
5. Reporting: Review the output, which includes a summary of detected vulnerabilities, failure rates, and examples of problematic responses. Use these findings to enforce stricter content filters or update system prompts.
Linux/Windows Commands:
Linux/macOS pip install garak garak --model_type huggingface --model_name "meta-llama/Llama-2-7b-chat-hf" --probes all --output json Windows (PowerShell) pip install garak garak --model_type openai --model_name "gpt-3.5-turbo" --probes "jailbreak" --verbose
3. CI/CD Integration for Security Testing
Modern DevSecOps practices demand that security testing becomes an integral part of the software development lifecycle. The third repository provides pre-built checks for prompts, agents, and RAG pipelines, enabling automated security scanning on every code commit. This proactive approach ensures that vulnerabilities are identified and remediated before they reach production environments.
Step‑by‑step guide explaining what this does and how to use it:
1. Integration: Include the security testing library in your `requirements.txt` or package.json.
2. Configuration: Define a security policy file (e.g., security_policy.yaml) that specifies prohibited content categories, acceptable toxicity thresholds, and allowed actions.
3. Pipeline Setup: Add a step to your CI/CD pipeline (e.g., GitHub Actions, Jenkins) that invokes the security scanner. For example, in GitHub Actions, create a workflow that runs `security_scan.py` against all updated prompt templates.
4. Execution: During the pipeline run, the scanner will analyze prompts and agent configurations, flagging any violations of the defined policy.
5. Reporting and Blocking: Configure the scanner to fail the build if critical vulnerabilities are detected, ensuring that insecure code cannot be merged.
Example CI/CD Snippet (GitHub Actions):
name: AI Security Scan on: [bash] jobs: security-scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Run LLM Security Checks run: | pip install llm-security-scanner llm-security-scanner --path ./prompts/ --policy ./policy.yaml
4. Runtime Protection with Prompt Sanitization
Runtime protection is the last line of defense against malicious inputs. The Protect AI toolkit—Vigil—offers a robust solution for prompt sanitization, PII detection, and harmful-content filtering. Vigil acts as a security middleware between the user input and the LLM, automatically rewriting or blocking dangerous inputs in real-time.
Step‑by‑step guide explaining what this does and how to use it:
1. Deployment: Deploy Vigil as a sidecar container or a standalone microservice using Docker.
2. Integration: Configure your application to route all prompts through Vigil before they reach the LLM. This can be achieved by modifying your API gateway or using the Vigil Python client.
3. Rule Configuration: Define rules using a YAML file to specify which patterns to sanitize (e.g., regex for social security numbers, blocklists for profanity, custom patterns for PII).
4. Operation: Vigil processes the prompt, removes or redacts sensitive information, and returns a sanitized version to the application. For blocked prompts, it can return a generic error response.
5. Monitoring: Monitor the filtering logs to track false positives and refine the sanitization rules over time.
Linux/Windows Commands:
Linux/macOS
docker run -p 5000:5000 protectai/vigil
curl -X POST http://localhost:5000/sanitize -H "Content-Type: application/json" -d '{"prompt": "My email is [email protected]"}'
Windows (PowerShell)
docker run -p 5000:5000 protectai/vigil
Invoke-WebRequest -Uri http://localhost:5000/sanitize -Method Post -Body '{"prompt": "My SSN is 123-45-6789"}' -ContentType "application/json"
5. AI Adversarial Defense with IBM’s ART
The Adversarial Robustness Toolbox (ART) from IBM is an essential resource for understanding and mitigating classical adversarial machine learning threats, such as evasion attacks, poisoning, and model extraction. While many modern attacks focus on prompt-level manipulation, ART addresses the foundational ML security issues that affect vision, tabular, and text models. This library is critical for organizations building custom models or fine-tuning existing ones on sensitive data.
Step‑by‑step guide explaining what this does and how to use it:
1. Installation: Install ART via `pip install adversarial-robustness-toolbox`.
- Model Wrapping: Wrap your trained TensorFlow, PyTorch, or scikit-learn model with ART’s `Classifier` interface to make it compatible with the toolbox’s attack and defense algorithms.
- Attack Simulation: Use ART’s attacks (e.g., Fast Gradient Sign Method, Projected Gradient Descent) to generate adversarial examples and evaluate your model’s susceptibility.
- Defense Application: Implement defenses such as defensive distillation, adversarial training, or input preprocessing (e.g., feature squeezing) to harden the model against the identified attack vectors.
- Validation: Re-evaluate the model with the same attacks to measure the effectiveness of your defenses and iterate accordingly.
Python Code Snippet:
from art.estimators.classification import PyTorchClassifier from art.attacks.evasion import FastGradientMethod import torch Wrap a PyTorch model model = ... your model criterion = torch.nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=0.01) classifier = PyTorchClassifier(model=model, loss=criterion, optimizer=optimizer, input_shape=(3, 32, 32), nb_classes=10) Generate adversarial examples attack = FastGradientMethod(estimator=classifier, eps=0.2) x_test_adv = attack.generate(x_test)
6. Guardrails for Conversational AI
Guardrails is a framework designed to define and enforce policies for conversational AI, specifying what a chatbot can and cannot do. It acts as a governance layer, enforcing rules about behavior, topic containment, and action execution. For example, you can enforce that a customer service bot never discusses stock trading or that it must always redirect to a human agent for refund requests.
Step‑by‑step guide explaining what this does and how to use it:
1. Installation: Install the Guardrails library: pip install guardrails-ai.
2. Creating Guards: Define guards using a configuration file or inline code. Guards consist of validators that check input and output.
3. Integrating with AI: Wrap your LLM call with a guard. When a user sends a prompt, the guard intercepts it, validates it against the policy, and either passes it through, modifies it, or denies it.
4. Handling Violations: Configure how violations are handled—for example, by returning a fixed error message, logging the incident, or triggering an alert.
5. Testing: Thoroughly test guard behavior with adversarial inputs to ensure they are not overly restrictive or too permissive.
Python Code Snippet:
import guardrails as gd from guardrails.validators import ValidLength, ValidRange Define a guard with a validator guard = gd.Guard.from_string( "The assistant must provide a response under 100 words.", validators=[ValidLength(max=100)] ) Use the guard with an LLM call response = guard(llm_call, prompt="Tell me about AI security.")
7. Understanding Indirect Prompt Injection
Indirect prompt injection is a critical attack vector where an attacker embeds malicious instructions into data that an LLM retrieves—for example, hidden in a web page or a document. This repository provides the original research and code to understand and exploit this vulnerability, serving as a mandatory reference for builders of AI agents that interact with external data sources.
Step‑by‑step guide explaining what this does and how to use it:
1. Setup: Clone the repository and inspect the proof-of-concept code.
2. Simulation: Run the provided scripts to simulate an indirect prompt injection attack, understanding how context can be poisoned by external content.
3. Mitigation: Implement mitigations such as sanitizing retrieved data, separating instructions from context, or using a dedicated model to classify safe vs. unsafe inputs.
What Undercode Say:
- Automated Red Teaming is Non-1egotiable: Manual prompt testing is insufficient. Frameworks like PyRIT allow teams to systematically explore the attack surface and identify failure modes at scale, which is essential for robust AI security.
- Runtime Protections Are the Safety Net: While proactive scanning and red-teaming are critical, they cannot catch every zero-day vulnerability. Solutions like Vigil and Guardrails provide runtime enforcement, acting as the final gatekeeper against malicious inputs and preventing data leakage at the moment of interaction.
Analysis: The confluence of these tools—covering red teaming, vulnerability scanning, CI/CD integration, runtime protection, adversarial ML, and research—represents a mature and comprehensive approach to AI security. The shift from reactive patching to proactive, automated security testing mirrors the evolution of software security in the early 2000s, but accelerated for the generative AI era. Organizations that embed these tools into their MLOps and DevOps pipelines will not only reduce their risk profile but will also foster a security-first culture that is essential for maintaining user trust and regulatory compliance. However, the rapid pace of innovation in adversarial techniques means that maintaining and updating these toolchains is as critical as their initial deployment.
Prediction:
- +1 The commoditization of AI red-teaming tools will democratize security, enabling smaller organizations to achieve enterprise-grade AI safety without massive internal expertise.
- -1 The increasing use of agentic systems that combine multiple LLMs and external tools will create compound vulnerabilities, making single-point security checks inadequate and necessitating holistic, cross-application security models.
- +1 Standardization of security evaluation metrics (e.g., robust benchmark suites) will emerge, allowing for objective comparison and selection of more secure foundational models and frameworks.
- -1 As defenses become widespread, attackers will shift to more subtle and persistent threats, such as data poisoning attacks against RAG pipelines and long-term adversarial prompting, requiring even more advanced continuous monitoring and adaptive defense strategies.
▶️ Related Video (84% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Andrii D – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
