Listen to this Post
Introduction:
The software development landscape is undergoing a seismic shift as Large Language Models (LLMs) like GitHub Copilot and ChatGPT become primary code generators. However, as highlighted by ProjectDiscovery’s CTO, the code produced by these AI models is alarmingly vulnerable right out of the box. If organizations integrate this code without rigorous security review, they are unknowingly shipping critical flaws. This article explores the risks of AI-generated code, providing a technical guide to identifying, exploiting, and mitigating these common vulnerabilities using open-source tools and manual review techniques.
Learning Objectives:
- Understand the inherent security weaknesses in LLM-generated code and the concept of “shipping unknown vulnerabilities.”
- Learn how to integrate automated security scanning tools (like Nuclei and Semgrep) into a CI/CD pipeline to catch AI-generated flaws.
- Master a hybrid approach combining AI-driven analysis with human-led architecture and edge-case reviews for critical systems.
You Should Know:
- The Anatomy of Vulnerable AI Code: Setting Up Your Lab
Before we can fix vulnerabilities, we need to understand what they look like. LLMs are trained on vast public datasets, which include insecure code snippets from forums and legacy repositories. Consequently, they often generate code susceptible to SQL Injection, Cross-Site Scripting (XSS), and Insecure Direct Object References (IDOR).
To test this, let’s create a simple Python Flask endpoint that an AI might generate for a user profile feature.
Step‑by‑step guide:
Create a file named `app.py`:
from flask import Flask, request
import sqlite3
app = Flask(<strong>name</strong>)
Example of potentially vulnerable AI-generated code
@app.route('/profile')
def profile():
Get user ID directly from the request without validation
user_id = request.args.get('id')
Connect to database
conn = sqlite3.connect('users.db')
cursor = conn.cursor()
VULNERABILITY: Direct string formatting leads to SQL Injection
query = f"SELECT FROM users WHERE id = {user_id}"
cursor.execute(query)
user = cursor.fetchone()
conn.close()
return f"User Data: {user}"
if <strong>name</strong> == '<strong>main</strong>':
app.run(debug=True)
What this does: This code takes a user-supplied `id` parameter and plugs it directly into an SQL query. An attacker can manipulate this parameter to dump the entire database. This is a classic example of an LLM replicating insecure patterns.
- Dynamic Scanning with Nuclei: The First Line of Defense
ProjectDiscovery’s Nuclei is an excellent tool for the first pass of security review. It sends requests to your application based on templates to detect known vulnerabilities.
Step‑by‑step guide to test the vulnerable endpoint:
1. Install Nuclei: (Assuming you have Go installed)
go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest
2. Run a SQL Injection Scan: Target your locally running Flask app (e.g., `http://localhost:5000`).
nuclei -u http://localhost:5000/profile?id=1 -t exposures/configs/ -t vulnerabilities/ -tags sqli
3. Manual Exploitation for Verification:
Use `curl` or `sqlmap` to confirm the flaw.
Using curl to test for basic error-based SQLi curl "http://localhost:5000/profile?id=1'" Using sqlmap for automated exploitation sqlmap -u "http://localhost:5000/profile?id=1" --batch --dbs
What this does: Nuclei acts as an automated “human” reviewer, sending malicious payloads to the endpoint. The manual commands verify that the AI’s code is indeed vulnerable, demonstrating why this first-pass automation is critical before code ever reaches production.
- Static Analysis with Semgrep: Hunting for Insecure Patterns
While dynamic scanning tests the running app, static analysis reviews the code structure itself. This is where we can catch the “logic” flaws that LLMs often miss, such as hardcoded credentials or weak cryptography.
Step‑by‑step guide:
1. Install Semgrep:
python3 -m pip install semgrep
2. Run a Scan on the `app.py` file:
Semgrep has specific rules for finding dangerous string formatting in SQL queries.
semgrep --config "p/owasp-top-ten" app.py
3. Review the Output: Semgrep will highlight the line `query = f”SELECT FROM users WHERE id = {user_id}”` and flag it as a potential SQL injection (based on rule python.lang.security.audit.formatted-sql-query.formatted-sql-query).
4. Writing a Custom Rule for AI Hallucinations: You can create a custom rule to detect if an AI has generated placeholder credentials.
Create a file `custom.yml`:
rules: - id: hardcoded-ai-credential pattern-either: - pattern: 'password = "CHANGE_ME"' - pattern: 'api_key = "sk-..."' message: "Hardcoded or placeholder credential detected." languages: [bash] severity: WARNING
Run it with: `semgrep –config custom.yml app.py`
What this does: Semgrep integrates directly into your IDE or CI pipeline, acting as the “AI code reviewer” for the AI itself. It ensures that the insecure patterns the LLM learned are flagged before a human even looks at the pull request.
4. Mitigation: Secure Code Generation and Parameterized Queries
Once vulnerabilities are identified, the fix must be applied. You should never trust the LLM to fix its own code without human oversight, as it may introduce new flaws.
Step‑by‑step guide to fix the SQL Injection:
Replace the vulnerable query section in `app.py` with a parameterized query using the DB-API.
Secure version
@app.route('/profile')
def profile():
user_id = request.args.get('id')
Input validation: Ensure it's a number
if not user_id or not user_id.isdigit():
return "Invalid user ID", 400
conn = sqlite3.connect('users.db')
cursor = conn.cursor()
SECURE: Use parameterized queries
cursor.execute("SELECT FROM users WHERE id = ?", (user_id,))
user = cursor.fetchone()
conn.close()
return f"User Data: {user}"
What this does: This code validates the input (ensuring it’s a digit) and separates the SQL logic from the data by using a placeholder (?). The database engine treats the `user_id` as data, not executable code, effectively neutralizing injection attempts. This is the type of architectural fix that AI often misses but human review must enforce.
- Infrastructure as Code (IaC) Hardening for AI-Deployed Apps
If an LLM is used to generate Terraform or CloudFormation scripts, it often defaults to the least secure configuration (e.g., public S3 buckets, open security groups). We need to scan for these.
Step‑by‑step guide using `checkov` to scan IaC:
1. Install Checkov:
pip install checkov
2. Create a vulnerable Terraform file (`main.tf`):
resource "aws_s3_bucket" "ai_generated_bucket" {
bucket = "my-ai-app-data"
acl = "public-read" Vulnerable: World-readable bucket
}
3. Scan it:
checkov -f main.tf
4. Interpret the Results: Checkov will output a `FAIL` for `CKV_AWS_20` (S3 Bucket has an ACL defined which allows public READ access.), forcing the developer to change the `acl` to private.
What this does: This command-line tool acts as a security linter for your cloud infrastructure. It catches the “out of the box” insecure defaults that an LLM might generate, ensuring the deployment environment is as hardened as the application code.
What Undercode Say:
- Automation is a filter, not a replacement: While tools like Nuclei and Semgrep are essential for catching the low-hanging fruit of LLM-generated flaws, they cannot assess business logic or architectural risk. High-stakes changes (auth, payment, infra) require human oversight to ensure the AI isn’t creating a logical backdoor. The biggest mistake is treating AI review tools as a final authority rather than a first-pass filter.
- You are shipping the training data’s debt: LLMs generate code based on past internet data, which includes decades of insecure practices. By accepting this code blindly, organizations are accelerating the deployment of technical debt and vulnerabilities. The “shift-left” movement must now account for “shift-left-AI,” where security review happens not just during coding, but during the prompt engineering phase to guide the LLM toward secure defaults.
Prediction:
The next major evolution in Application Security (AppSec) will be the rise of “LLM Firewalls” and adversarial prompt engineering for code generation. We will see tools emerge that intercept prompts to developers’ IDEs, rewriting insecure requests (e.g., “write a login script”) into secure, parameterized templates before the AI even generates a single line of code. The battleground will shift from reviewing code to securing the prompts that create the code.
▶️ Related Video (72% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Ehsandeepsingh This – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
