Listen to this Post
Introduction:
When Cisco released Antares—small, open-weight models designed to localize vulnerabilities locally so customer source code never leaves the building—the security community applauded the privacy-first approach. However, putting an AI model in charge of identifying code flaws introduces a critical trust paradox: how do you verify the verifier? In a weekend experiment that systematically tested Antares against real-world vulnerability data, researchers discovered that asking an AI to hunt bugs doesn’t always mean it’s actually hunting—and sometimes, it just invents the evidence.
Learning Objectives:
- Understand the operational limitations of AI-powered vulnerability localization tools and when they can be trusted
- Learn how to design test harnesses that distinguish between model failures and implementation errors
- Master techniques for extracting verifiable ground truth from public security advisories and commit histories
You Should Know:
- The Path Traversal Misidentification: When the Model Chose the Wrong File
The experiment began with a straightforward test: feed Cisco’s security model a CWE class (path traversal) and see if it could identify the vulnerable file in a known, already-fixed repository. The model pointed confidently atpathconverter.py. The actual bug existed inb64.py. This fundamental mismatch reveals a critical issue: the model matched the filename’s semantic context rather than analyzing the actual code. It saw “path” and guessed “pathconverter,” ignoring the base64 encoding logic that actually contained the traversal vulnerability.
Step-by-Step Guide: Reproducing a Path Traversal Detection Environment
To understand how this misidentification occurs, you need to replicate the testing environment. Here’s how to set up a controlled vulnerability validation pipeline:
Linux Environment Setup:
Clone a vulnerable repository at the pre-fix commit git clone https://github.com/example/vulnerable-repo.git cd vulnerable-repo git checkout <commit-hash-before-fix> Extract all files that were modified in the security patch git diff --1ame-only <commit-hash-before-fix> <commit-hash-fix> > fixed_files.txt Set up a Python virtual environment for the security model python3 -m venv antares-env source antares-env/bin/activate pip install transformers torch requests
Windows PowerShell Equivalent:
Clone repository git clone https://github.com/example/vulnerable-repo.git cd vulnerable-repo git checkout <commit-hash-before-fix> Get modified files git diff --1ame-only <commit-hash-before-fix> <commit-hash-fix> | Out-File fixed_files.txt Set up Python virtual environment python -m venv antares-env .\antares-env\Scripts\activate pip install transformers torch requests
Extracting Ground Truth for Testing:
import os
import json
from pathlib import Path
def load_security_advisory(cve_id):
"""Load CVE data and extract affected files/modules"""
Example: fetch from NVD API
import requests
response = requests.get(f"https://services.nvd.nist.gov/rest/json/cves/2.0?cveId={cve_id}")
data = response.json()
Parse configurations and vulnerable software
return data
def create_answer_key(repo_path, fix_commit):
"""Generate ground truth from git commits"""
import subprocess
result = subprocess.run(
['git', 'diff', '--1ame-only', fix_commit],
cwd=repo_path,
capture_output=True,
text=True
)
return result.stdout.strip().split('\n')
- The “Clean” Conundrum: When No Vulnerability Becomes a False Positive
Perhaps the most alarming finding was the model’s inability to report “there is nothing here”—a behavior explicitly documented in its own model card. The researchers tested this on clean repositories, with a bigger budget, and even when they explicitly reminded the model that reporting “no vulnerabilities” was an option, it never took it. Under pressure to find something absent, it invented a plausible filename and asked for it twenty-two times.
Testing for False Positives: Building a Clean Repository Validator
def validate_clean_repository(repo_path, model_client, max_iterations=25):
"""
Test whether the model can correctly identify a clean repository
and terminate with a "no vulnerabilities found" verdict
"""
files = collect_source_files(repo_path)
for iteration in range(max_iterations):
prompt = create_security_prompt(files, require_determination=True)
response = model_client.query(prompt)
if "no vulnerabilities" in response.lower() or "clean" in response.lower():
print(f"✅ Model correctly reported clean at iteration {iteration+1}")
return True
If model invents files, track this behavior
if "please provide file" in response.lower():
invented_file = extract_filename(response)
print(f"⚠️ Model invented file: {invented_file} at iteration {iteration+1}")
print("❌ Model failed to report clean after maximum iterations")
return False
Detecting Model Hallucination Through Tool Call Counters:
The critical insight here is that the model’s transcript showed nineteen tool calls reaching the sandbox—except a parser bug meant zero tool calls ever actually executed. This is the difference between reading the transcript and reading the counters. Security practitioners must build monitoring that separates the model’s “claimed” actions from actual system-level executions.
- The Harness Bug That Changed Everything: Regular Expressions vs. Reality
Three times, what appeared to be model weakness turned out to be harness implementation errors. The most egregious: a parser bug caused zero tool calls to reach the sandbox, while the transcript showed nineteen. This looked exactly like a small model stuck in a loop—but it was actually a regular expression silently dropping tool calls.
Step-by-Step: Validating Your AI Security Toolchain Harness
1. Implement Tool Call Verification:
import re
import hashlib
from dataclasses import dataclass
from typing import List, Optional
@dataclass
class ToolCall:
tool_name: str
parameters: dict
call_id: str
executed: bool = False
result: Optional[bash] = None
class ToolCallMonitor:
"""Separate transcript from actual execution"""
def <strong>init</strong>(self):
self.tool_calls = {}
self.execution_log = []
self.transcript_log = []
def parse_transcript(self, model_output: str) -> List[bash]:
"""Extract tool calls from model transcript"""
WARNING: This regex is what caused the bug - needs validation
pattern = r'<tool_call>\s({.?})\s</tool_call>'
matches = re.findall(pattern, model_output, re.DOTALL)
tool_calls = []
for match in matches:
try:
This is where the parsing bug happened - invalid JSON
due to malformed tool calls would silently drop
call_data = json.loads(match)
tool_calls.append(ToolCall(
tool_name=call_data.get('name', 'unknown'),
parameters=call_data.get('parameters', {}),
call_id=hashlib.md5(match.encode()).hexdigest()[:8]
))
except json.JSONDecodeError:
BUG: This silently dropped calls instead of logging
print(f"⚠️ Failed to parse tool call: {match[:50]}...")
return tool_calls
def execute_tool_call(self, tool_call: ToolCall) -> bool:
"""Actually run the tool and verify it touched the sandbox"""
try:
Simulate sandbox execution
if tool_call.tool_name == "read_file":
Check if file exists before reading
file_path = tool_call.parameters.get('file_path')
if not os.path.exists(file_path):
tool_call.result = "File not found"
tool_call.executed = False
self.execution_log.append(f"❌ Failed: {tool_call.call_id}")
return False
Read the actual file
with open(file_path, 'r') as f:
tool_call.result = f.read()
tool_call.executed = True
self.execution_log.append(f"✅ Executed: {tool_call.call_id}")
return True
except Exception as e:
tool_call.result = f"Error: {str(e)}"
tool_call.executed = False
self.execution_log.append(f"❌ Error: {tool_call.call_id} - {str(e)}")
return False
def compare_transcript_to_execution(self):
"""Find discrepancies between claimed and actual tool calls"""
transcript_ids = {call.call_id for call in self.transcript_log}
execution_ids = {call.call_id for call in self.execution_log}
missing = transcript_ids - execution_ids
if missing:
print(f"🚨 Tool calls claimed but never executed: {missing}")
print(" (This indicates a harness parser bug)")
return False
return True
2. Create a Validation Harness Test:
!/bin/bash
test_harness.sh - Validate your toolcall parsing
echo "Starting harness validation..."
Create a test tool call that should be parsable
echo '{"name": "read_file", "parameters": {"file_path": "/etc/hosts"}}' > test_call.json
Attempt parsing with your current implementation
python -c "
import json
with open('test_call.json', 'r') as f:
data = json.load(f)
assert data['name'] == 'read_file', 'Parsing failed'
print('✅ Valid tool call parsed correctly')
" || echo "❌ Parsing bug detected - tool calls will be silently dropped"
Check for silent failures
echo "Checking for silent failures in tool call pipeline..."
python -c "
import re, json
This simulates the bug pattern
bad_input = '<tool_call>{\"name\": \"read_file\"}</tool_call>' Missing parameters
try:
The buggy approach that caused the issue
pattern = r'<tool_call>\s({.?})\s</tool_call>'
match = re.search(pattern, bad_input)
if match:
json.loads(match.group(1)) This will fail on malformed JSON
print('✅ Parsed')
else:
print('❌ No match - silent drop')
except json.JSONDecodeError:
print('❌ JSON decode error - tool call silently dropped')
"
- Building the Operational Envelope: Where the Model Can Be Trusted
The core question isn’t “is this model accurate?” but rather “where can this be trusted, and where can it not?” The researchers identified key limitations: eight test cases is not a statistically meaningful rate, the corpus favored small localized fixes, the harness was stricter than the vendor’s own CLI, and they tested the smallest model in the family.
Step-by-Step: Creating Your Own AI Security Model Operating Envelope
1. Define Trust Boundaries:
class OperationalEnvelope:
def <strong>init</strong>(self):
self.boundaries = {
'repository_size': {'max': 100000, 'unit': 'lines'},
'vulnerability_types': ['path_traversal', 'xss', 'sqli'],
'confidence_threshold': 0.75,
'max_iterations': 10
}
def assess_fit(self, context):
"""Determine if the model is operating within known boundaries"""
score = 0
if context['repo_size'] < self.boundaries['repository_size']['max']:
score += 25
if context['cwe_class'] in self.boundaries['vulnerability_types']:
score += 25
if context['iteration_count'] < self.boundaries['max_iterations']:
score += 25
if context['confidence'] > self.boundaries['confidence_threshold']:
score += 25
return score
2. Generate Confidence Metrics from Validation Data:
Generate a validation dataset from known CVEs
python -c "
import requests
from datetime import datetime, timedelta
Fetch recent CVEs with known fixes
endpoint = 'https://services.nvd.nist.gov/rest/json/cves/2.0'
params = {
'pubStartDate': (datetime.now() - timedelta(days=30)).isoformat(),
'resultsPerPage': 100
}
response = requests.get(endpoint, params=params)
cves = response.json().get('vulnerabilities', [])
Extract repositories and fix commits
for cve in cves:
cve_id = cve['cve']['id']
Look for references to GitHub commits
for ref in cve['cve'].get('references', []):
url = ref.get('url', '')
if 'github.com' in url and '/commit/' in url:
print(f'Valid test case: {cve_id} - {url}')
" > validation_cases.txt
3. Implement a Sandbox Validation Layer:
Create a validation script that separates transcript from actual execution cat > validate_model_response.sh << 'EOF' !/bin/bash Validates that model tool calls actually reached the sandbox MODEL_OUTPUT="$1" EXECUTION_LOG="$2" echo "Validating model claims vs actual execution..." Extract claimed tool calls from transcript claimed_calls=$(grep -o '<tool_call>.</tool_call>' "$MODEL_OUTPUT" | wc -l) actual_calls=$(wc -l < "$EXECUTION_LOG") echo "Tool calls claimed: $claimed_calls" echo "Tool calls executed: $actual_calls" if [ "$claimed_calls" -1e "$actual_calls" ]; then echo "❌ TRANSCRIPT/EXECUTION MISMATCH DETECTED" echo " This indicates a harness parser bug or model hallucination" exit 1 else echo "✅ All claimed tool calls were actually executed" exit 0 fi EOF chmod +x validate_model_response.sh
What Undercode Say:
- The file name matching problem is more dangerous than raw accuracy metrics—when a security tool points at `pathconverter.py` but the bug is in
b64.py, it’s not just a false negative but a fundamental failure of code understanding that could lead security teams down the wrong rabbit hole -
The inability to report “clean” creates a catastrophic failure mode where noisy, low-confidence findings are indistinguishable from accurate ones—an AI that always says “maybe something here” is worse than one that occasionally misses vulnerabilities because it erodes trust across the board
-
Tool call hallucination at 22 iterations reveals that under pressure, the model generates plausible-sounding requests rather than admitting defeat—this behavior of inventing filenames and repeatedly requesting them is a form of “performance anxiety” that’s documented in the model card but underestimated in practice
-
The parser bug causing zero tool calls despite 19 showing in the transcript is the most critical lesson—it’s not enough to audit the model’s output; you must audit the entire pipeline including your own harness, as implementation errors can mask as model failures
-
Eight test cases is not a sufficient sample size for any statistically meaningful conclusion—the researchers knew this and framed their findings as an operational envelope rather than a benchmark, which is the honest approach
-
The difference between deploying a model and governing one lies in the boring but decisive artifact: the operating envelope—where the thing works, where it fails, and how you would know the difference
Prediction:
+1: The demand for AI security evaluation frameworks that include harness validation will create a new market for “model governance tooling” beyond simple vulnerability scanners, similar to how penetration testing evolved from basic port scanning to comprehensive adversarial simulation
-1: The documented hallucination behavior where models invent filenames when under pressure will become a more significant liability than accuracy metrics—as organizations automate remediation based on AI findings, these hallucinations could trigger unnecessary incident response or, worse, lull teams into false security when no vulnerabilities exist
+1: The repository methodology using published advisories with real fix commits as ground truth will become a standard practice for evaluating AI security models, replacing synthetic benchmarks that don’t capture the complexity of real-world codebases
-1: Until there’s a standardized way to validate that AI security tools can actually report “nothing found,” organizations will continue to ignore these tools in favor of traditional SAST/DAST—the inability to distinguish “no vulnerabilities” from “I didn’t find any” creates a trust gap that data-driven security teams are unwilling to bridge
+1: The honest disclosure of limitations—eight cases, smallest model, stricter harness—demonstrates a maturity in AI security discourse that will benefit the industry, as it moves away from benchmark-fishing to operational clarity about what these systems can and cannot actually do
The critical insight for security practitioners: AI security models are not magic oracle boxes—they’re complex systems with known failure modes that need governance, validation, and calibration against ground truth. The question isn’t “does it work?” but “where does it work, where does it not, and how would you know?” That operating envelope is the artifact that separates deploying a model from governing one.
▶️ Related Video (70% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Osramek Aisecurity – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
