Listen to this Post
Introduction:
The cybersecurity industry has long debated whether large language models (LLMs) can truly function as autonomous penetration testers or if they are merely sophisticated pattern-matchers. A groundbreaking benchmark has now put this question to the test, pitting 15 different LLMs against OWASP Juice Shop—a deliberately vulnerable web application—under identical, no-hand-holding conditions. The results have upended conventional wisdom, revealing that unexpected models from China and xAI have outperformed established Western frontier models in both speed and vulnerability discovery, while doing so at a fraction of the cost.
Learning Objectives:
- Understand the methodology and significance of head-to-head LLM penetration testing benchmarks against standardized targets like OWASP Juice Shop.
- Analyze the performance metrics—vulnerability count, speed, and cost-per-finding—that differentiate leading AI models in offensive security roles.
- Learn how to set up and execute automated LLM-driven security testing using open-source tools and harnesses.
- Identify practical Linux, Windows, and API security commands to replicate and verify LLM-discovered vulnerabilities.
- Evaluate the implications of open-weight models like GLM-5.2 for the future of accessible, cost-effective cybersecurity.
You Should Know:
1. The Benchmark That Rewrote the Leaderboard
The benchmark was deliberately designed to eliminate bias and prompt engineering tricks. Each of the 15 LLMs was given the same target (OWASP Juice Shop), the same tools, and the same sandboxed environment. The models received source code access and a running instance, with a simple instruction: “find vulnerabilities.” No hand-holding, no embedded hints—just raw, autonomous discovery.
The results, compiled after 38 runs, were striking. Kimi K3 from Moonshot AI led the pack with 62 vulnerabilities discovered, surpassing every Claude and GPT model tested. Grok 4.5 from xAI secured second place with 59 findings, but its true distinction was speed: it completed the task in just 26 minutes, offering the best speed-to-quality ratio of any model. Claude Opus 4.5 took third place with 57 findings, but in a shocking twist, its newer flagship variant, Opus 4.8, plummeted to 11th place with only 26 findings—demonstrating that newer is not always better in AI security capabilities.
Perhaps the most disruptive revelation came from GLM 5.2 (Zhipu AI), which achieved the lowest cost per finding at just $0.007—less than a penny per vulnerability. This economic advantage, confirmed by independent researchers, positions open-weight Chinese models as formidable challengers to expensive proprietary U.S. alternatives.
2. Setting Up Your Own LLM Pentesting Harness
To replicate this type of benchmark, you need an automated harness that orchestrates the LLM, executes commands, and validates findings. The TrustedSec benchmark provides a minimalistic yet effective blueprint.
Step-by-step guide:
Step 1: Deploy the Target
Pull and run OWASP Juice Shop in a Docker container docker pull juice-shop/juice-shop docker run -d -p 3000:3000 juice-shop/juice-shop
The application will be accessible at `http://localhost:3000`.
Step 2: Set Up the LLM Backend
For self-hosted models, use Ollama to serve the LLM via an OpenAI-compatible API:
Install Ollama and pull a model (e.g., Llama 3 or Mistral) curl -fsSL https://ollama.com/install.sh | sh ollama pull llama3:latest ollama serve
The harness will communicate with Ollama’s API endpoint (typically `http://localhost:11434/v1`).
Step 3: Build the Harness Logic
The harness should:
- Build a message history with a system prompt (e.g., “You are a penetration tester.”)
- Provide the LLM with tools: typically an `http_request` function and an `encode_payload` function
- Execute tool calls against the target and feed results back to the model
- Check each HTTP response against success conditions (e.g., challenge completion flags)
- Loop until success, turn limit, or error
Step 4: Run the Benchmark
Execute multiple independent runs per model to reduce variance. The TrustedSec study recommends 100 runs per challenge for statistical significance.
3. Linux Commands for Vulnerability Verification
When an LLM flags a potential vulnerability, you must verify it manually. Here are essential Linux commands for common Juice Shop vulnerability classes:
SQL Injection Testing
Basic boolean-based blind SQLi test curl -X GET "http://localhost:3000/rest/products/search?q=' OR '1'='1' --%" Time-based blind SQLi curl -X GET "http://localhost:3000/rest/products/search?q=sleep(5)--"
Broken Access Control (IDOR) Testing
Attempt to access another user's basket curl -X GET "http://localhost:3000/rest/basket/2" -H "Authorization: Bearer <your_token>" Check if you can view another user's profile curl -X GET "http://localhost:3000/rest/user/profile" -H "Authorization: Bearer <stolen_token>"
GLM-5.2 achieved a 39% F1 score specifically on IDOR detection, outperforming Claude Code’s 32-37% range in some evaluations.
XSS Payload Testing
Reflected XSS test
curl -X GET "http://localhost:3000/rest/products/search?q=<script>alert('XSS')</script>"
Stored XSS via feedback form
curl -X POST "http://localhost:3000/api/Feedbacks" -H "Content-Type: application/json" -d '{"comment":"<script>alert(1)</script>","rating":5}'
4. Windows Commands for API Security Testing
For Windows environments, PowerShell and the native `curl` alias provide similar capabilities:
JWT Manipulation
Juice Shop uses JSON Web Tokens (JWT) for authentication. Weak JWT implementation is a common finding:
Decode a JWT token (base64 decode the payload)
$token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InRlc3RAdGVzdC5jb20ifQ.signature"
$payload = $token.Split('.')[bash]
Add padding and decode
$payload += '=' (4 - ($payload.Length % 4))
Testing for Weak JWT Secrets
Attempt to forge a token with a weak secret (e.g., "secret")
Use a tool like jwt_tool or custom Python script
python -c "import jwt; print(jwt.encode({'email':'[email protected]'}, 'secret', algorithm='HS256'))"
CSRF Testing
Attempt a state-changing request without a CSRF token
curl -X POST "http://localhost:3000/api/Users" -H "Content-Type: application/json" -d '{"email":"[email protected]","password":"12345"}'
5. Tool Configurations and Automation
The benchmark compared multiple automated tools against Juice Shop. Here are configurations for key tools:
ZAP (Zed Attack Proxy)
Run ZAP in headless mode against Juice Shop zap.sh -cmd -quickurl http://localhost:3000 -quickprogress -quickout zap_report.html
ZAP found 593 findings but had a 46.6% false positive rate and solved only 2 challenges.
Nuclei
Run Nuclei templates against the target nuclei -u http://localhost:3000 -t ~/nuclei-templates/ -severity critical,high
Nuclei found only 1 finding but had a 0% false positive rate and solved 3 challenges.
pentest-ai (ptai)
Run the ptai sweep command ptai sweep -u http://localhost:3000
ptai found 88 vulnerabilities across 5 OWASP Top 10 categories, with 0% false positives—outperforming traditional scanners in quality.
6. Cloud Hardening and API Security Best Practices
LLM-discovered vulnerabilities often mirror real-world cloud misconfigurations. Implement these hardening measures:
API Rate Limiting
Nginx configuration to prevent brute-force attacks
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
location /rest/user/login {
limit_req zone=login burst=3 nodelay;
}
Input Validation and Parameterized Queries
// Node.js/Express - Avoid string concatenation in SQL queries
// Vulnerable (SQL injection)
const query = <code>SELECT FROM Products WHERE name LIKE '%${req.query.q}%'</code>;
// Secure (parameterized)
const query = 'SELECT FROM Products WHERE name LIKE ?';
db.all(query, [<code>%${req.query.q}%</code>], callback);
JWT Security
// Use strong secrets (minimum 32 characters)
const secret = process.env.JWT_SECRET || crypto.randomBytes(64).toString('hex');
// Set short expiration times
const token = jwt.sign({ email: user.email }, secret, { expiresIn: '15m' });
// Always verify the token on every request
jwt.verify(token, secret, (err, decoded) => { / handle / });
7. Vulnerability Exploitation and Mitigation Walkthrough
One of the most common Juice Shop vulnerabilities is Insecure Direct Object Reference (IDOR) in the basket functionality.
Exploitation Steps:
- Log in as user A and capture the JWT token.
- Change the basket ID in the URL: `GET /rest/basket/2` (attempting to access user B’s basket).
- If the application does not verify authorization, you can view and modify other users’ baskets.
Mitigation:
// Always check ownership
app.get('/rest/basket/:id', (req, res) => {
const basketId = req.params.id;
const userId = req.user.id; // from JWT
db.get('SELECT FROM Baskets WHERE id = ? AND UserId = ?', [basketId, userId], (err, basket) => {
if (!basket) return res.status(403).json({ error: 'Unauthorized' });
res.json(basket);
});
});
What Undercode Say:
- Key Takeaway 1: The AI penetration testing leaderboard is no longer dominated by the usual suspects. Kimi K3 and Grok 4.5 have demonstrated that unexpected models can outperform flagship offerings from Anthropic and OpenAI in specific security tasks, challenging the assumption that the most advanced general-purpose models are always the best for specialized cybersecurity work.
- Key Takeaway 2: Cost efficiency is now a decisive factor. GLM-5.2’s sub-penny cost per finding—and even the more conservative $0.17 per finding estimate from other studies—makes AI-assisted vulnerability discovery accessible to organizations that cannot afford expensive proprietary API calls. This democratization of security AI could level the playing field between well-funded enterprises and smaller teams.
- Analysis: The benchmark reveals a fundamental shift: the cybersecurity community must now evaluate LLMs not just on raw capability but on speed, cost, and reliability. The surprising underperformance of Claude Opus 4.8 compared to its predecessor Opus 4.5 highlights that model updates do not guarantee improvements in security-specific reasoning. Furthermore, the rise of open-weight Chinese models like GLM-5.2 raises geopolitical questions about AI export controls—if capable models are freely downloadable, restricting access to U.S. frontier models may no longer be an effective strategy.
Prediction:
- -1: The widening gap between LLM discovery capabilities and organizational patch management will create a “findings debt” crisis. As AI tools flood security teams with thousands of vulnerabilities, manual triage will become the bottleneck, potentially increasing the window of exposure for critical flaws.
- +1: The low cost of GLM-5.2 and similar open-weight models will spur innovation in automated remediation pipelines. We will see a surge in CI/CD-integrated “fix-as-you-code” agents that not only find vulnerabilities but also generate and test patches, reducing mean time to remediation from weeks to hours.
- +1: The benchmark’s transparent methodology will become an industry standard, forcing LLM providers to compete on verifiable security performance rather than marketing claims. This will accelerate the development of specialized security models optimized for specific vulnerability classes (e.g., IDOR, SQLi, XSS).
- -1: As LLM penetration testing becomes cheaper and more accessible, we will also see a corresponding rise in AI-powered offensive campaigns. Adversarial groups will leverage these same models to automate reconnaissance and exploit discovery, shifting the asymmetry of cyber warfare further toward attackers who can deploy AI at scale.
- +1: The integration of LLM-based testing into DevSecOps pipelines will significantly reduce the cost of security defects caught early in the development lifecycle, potentially saving the industry billions in breach-related damages annually.
▶️ Related Video (68% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Omer Talmy – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
