AI in AppSec: The 5% Certainty Problem and What It Means for Defenders + Video

By /

Listen to this Post

Featured Image

Introduction:

The application security landscape has reached an inflection point. According to Contrast Security’s AppSec Overflow 2026 report, adversaries now touch the average application once every four minutes, with 42 confirmed viable exploitation attempts per application per month. Meanwhile, three AI scanners analyzing the same codebase agreed on only 5 percent of findings—and a single scanner run three times against identical code reproduced just 17 percent of its own results. This non-determinism, combined with a patching backlog where critical vulnerabilities take an average of 92 days to remediate, signals that the traditional “find-and-fix” model is broken. As Jeff Williams, CTO of Contrast Security, put it: “AI ended that race, and defenders lost it”.

Learning Objectives & Secrets:

  • Objective 1: Master Vulnerability Prioritization Beyond CVSS — Learn to combine CVSS severity scores with EPSS (Exploit Prediction Scoring System) probability scores and CISA’s KEV (Known Exploited Vulnerabilities) catalog to build a risk-based triage model that moves beyond theoretical severity. Secret tip: Prioritize any CVE on the KEV list affecting your stack, any CVE with EPSS above 0.6 and reachable code paths, and any internet-exposed CVE with CVSS 9.0+.

  • Objective 2: Implement Runtime Defense as the Source of Truth — Traditional SAST, DAST, and AI scanners operate on static code or simulated traffic; they cannot tell you how your application behaves when actually attacked. Secret tip: Deploy Application Detection and Response (ADR) or Runtime Application Self-Protection (RASP) to observe attacks from inside the running application, distinguishing between bulk probes and viable exploits.

  • Objective 3: Automate Vulnerability Intelligence Feeds — Don’t rely solely on scanner outputs. Secret tip: Build automated pipelines that pull from the NVD API, OSV.dev, and CISA KEV to enrich and contextualize findings, reducing the noise that overwhelms security teams.

You Should Know:

  1. The Attack Surface Is Expanding Faster Than Defenders Can Patch

The numbers are sobering. Monitored applications carry an average of 106 vulnerability findings in custom code, including 22 rated high or critical severity. Development and security teams remediate only 3.4 vulnerabilities per application per month—a rate that fails to keep pace with new flaw creation. In third-party code, 54% of CVE instances observed in production come from CVEs published more than a year ago, with Spring4Shell (CVE-2022-22965) and Log4Shell (CVE-2021-44228) still widely present years after disclosure.

Step-by-Step Guide: Querying the NVD API for Vulnerability Intelligence

To build your own vulnerability intelligence pipeline, start by pulling real-time CVE data from the NIST National Vulnerability Database:

  1. Request a free NVD API key at https://nvd.nist.gov/developers/request-an-api-key

  2. Set the API key as an environment variable:

    export NVD_API_KEY="your-api-key-here"

  3. Query CVEs by keyword using Python and nvdlib:

    import nvdlib
r = nvdlib.searchCVE(cpeName='cpe:2.3:a:apache:log4j:2.14.1', limit=10)
for cve in r:
print(f"{cve.id} - {cve.score[bash].baseScore}")

  4. Fetch the full JSON feed for a given year (Linux/macOS):

    YEAR=2026
wget https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-$YEAR.json.gz
gunzip nvdcve-1.1-$YEAR.json.gz

5. Parse and filter high-severity CVEs using `jq`:

cat nvdcve-1.1-2026.json | jq '.CVE_Items[] | select(.impact.baseMetricV3.cvssV3.baseScore >= 9.0) | .cve.CVE_data_meta.ID'

This approach gives you programmatic access to the same vulnerability data that powers commercial scanners—but with full control over how you filter, prioritize, and operationalize it.

  1. AI Scanners Are Noisy, Non-Deterministic, and Expensively Misleading

Contrast Labs tested three AI scanning approaches against enterprise Java codebases. Approach 1 (Simple Mode) returned 3,560 findings from a 1.8-million-line codebase, with 1,000 flagged as high severity. The API cost was $315. The triage labor cost? Approximately $128,000. Worse, 49% of findings appeared in exactly one run, and AI reviewing its own findings confirmed just 1.1% as true positives. GPT-4 alone has been shown to produce false positive rates exceeding 50% in cryptographic code analysis.

Step-by-Step Guide: OSV-Scanner for Open Source Vulnerability Detection

Instead of relying solely on AI-powered scanners, integrate Google’s OSV-Scanner—which uses the curated OSV.dev database—into your CI/CD pipeline:

1. Install OSV-Scanner (Linux/macOS):

go install github.com/google/osv-scanner/cmd/osv-scanner@latest

2. Scan a project directory for dependency vulnerabilities:

osv-scanner scan -r ./my-project/

3. Scan a specific lockfile (package-lock.json, requirements.txt, etc.):

osv-scanner scan --lockfile ./package-lock.json

4. Scan a container image for vulnerable packages:

osv-scanner scan --image your-image:latest

5. Output results in JSON for further processing:

osv-scanner scan -r ./my-project/ --json > vulnerabilities.json
  1. Ignore specific vulnerabilities by creating an `osv-scanner.toml` config file: 
    [[bash]]
id = "CVE-2023-12345"
reason = "Not exploitable in our environment"

OSV-Scanner provides deterministic, reproducible results based on known, verified vulnerabilities—a stark contrast to the non-determinism of pure LLM-based scanners.

3. Vulnerability Prioritization Requires Layered Intelligence

CVSS severity scores alone are insufficient for prioritization. A CVSS 9.8 in an unreachable library may be lower priority than a CVSS 7.5 with active exploits in the wild. The Exploit Prediction Scoring System (EPSS) estimates the probability that a CVE will be exploited in the next 30 days. CISA’s KEV catalog documents vulnerabilities with confirmed active exploitation.

Step-by-Step Guide: Building a Risk-Based Prioritization Model

  1. Fetch the latest CISA KEV catalog (no API key required): 
    curl -s https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json > kev.json

2. Extract CVEs with known exploits:

cat kev.json | jq '.vulnerabilities[] | .cveID'

3. Query EPSS scores via the FIRST API:

curl -X POST https://api.first.org/epss/v1/cve \
-H "Content-Type: application/json" \
-d '{"cve": ["CVE-2024-12345"]}'
  1. Calculate a composite priority score (CVSS 40% + EPSS 60%): 
    priority_score = (normalized_cvss  0.4) + (epss_score  0.6)

5. Create a triage policy:

  • P0 (24–48 hours): Any CVE on KEV + reachable in your environment
  • P1 (1 week): EPSS > 0.5 + CVSS ≥ 7.0 + reachable
  • P2 (Monthly): High CVSS but low EPSS and not internet-exposed
  • P3 (Backlog): All other findings

This layered approach moves beyond the “severity theater” that leaves teams chasing the wrong vulnerabilities while real attacks succeed.

  1. Runtime Defense: The Only Way to Stop Attacks Before Patches Deploy

With critical vulnerabilities taking 92 days to patch and attackers weaponizing exploits in hours, the math doesn’t work. Contrast Security’s Application Detection and Response (ADR) platform embeds security instrumentation directly into running applications, blocking both automated and advanced threats at the function level. Unlike perimeter defenses (WAFs, firewalls), runtime defense observes attacks from inside the application, distinguishing between noise and genuine threats.

Step-by-Step Guide: Implementing Runtime Protection

  1. Deploy RASP/ADR agents into your application runtime (Java, .NET, Node.js, Python)—these instrument the application without code changes

  2. Configure attack blocking policies for top exploit techniques:

– Untrusted deserialization (the 1 confirmed exploit vector)
– Path traversal
– Method tampering
– SQL injection (consistently in the top five across all industries)

  1. Enable runtime reachability analysis to confirm whether a vulnerability is actually exploitable in your specific application context

  2. Integrate runtime telemetry with your SIEM or SOC for real-time alerting

  3. Use runtime data to validate and triage scanner findings—if a vulnerability isn’t reachable at runtime, it may not require immediate patching

The goal is not to eliminate scanning but to use runtime as the source of truth that separates signal from noise.

  1. Bug Bounty Programs Are Pulling Back—A Canary in the Coal Mine

HackerOne paused new submissions to the Internet Bug Bounty program in March 2026, and Node.js paused its own program shortly afterward. This reflects a broader shift: as AI makes vulnerability discovery easier for attackers, the economics of crowdsourced discovery are changing. The Zero Day Clock recorded that the mean time to exploit dropped from over two years in 2018 to below one year by 2021, and the majority of exploited vulnerabilities in 2025 were weaponized within three weeks.

What This Means for Your Organization:

  • Don’t rely on bug bounties as your primary discovery mechanism
  • Build internal red-team capabilities augmented by AI—but validate all findings manually
  • Shift from “find and fix” to “detect and block” at runtime
  • Invest in Application Detection and Response (ADR) as a compensating control while patches are developed

What Undercode Say:

  • Key Takeaway 1: AI security scanners are not a silver bullet—they are noisy, non-deterministic, and expensive to triage. The 5% agreement rate across three scanners and 17% self-consistency rate should be a red flag for any organization making these tools their single source of truth. AI can augment, but it cannot replace, human judgment and runtime validation.

  • Key Takeaway 2: The find-and-fix model is fundamentally broken when attackers weaponize vulnerabilities in hours and defenders take 92 days to patch. The only sustainable defense is runtime protection that blocks attacks while patches are developed—not scanning harder or hiring more people.

The data from Contrast’s AppSec Overflow 2026 report paints a clear picture: the application security industry is at a crossroads. AI is accelerating both attack and defense, but the defenders are losing the race. The path forward requires a fundamental shift—from treating vulnerability scanning as the primary control to embracing runtime defense as the new source of truth. Organizations that continue to rely solely on static and AI-based scanning will find themselves buried in noise while real attacks succeed. Those that adopt runtime visibility, layered prioritization models, and automated intelligence feeds will have a fighting chance.

Prediction:

  • -1 The 5% AI scanner agreement rate will not improve significantly in the next 12–18 months—LLM architecture non-determinism is inherent, and vendors will prioritize feature velocity over accuracy.

  • -1 The average time to exploit will continue to drop below current levels as AI-powered attack tools become more accessible, compressing the window between disclosure and weaponization.

  • +1 Runtime Application Detection and Response (ADR) will become the mandatory standard for production applications within 24–36 months, driven by both regulatory pressure and the sheer impossibility of patching fast enough.

  • -1 Bug bounty programs will face continued contraction as the economics of crowdsourced discovery become unsustainable in an AI-driven threat landscape.

  • +1 Organizations that build automated vulnerability intelligence pipelines (NVD + EPSS + KEV) will achieve 60-70% reduction in mean time to remediation by focusing on the 5% of vulnerabilities that actually matter.

  • -1 The “AI scanner tax”—$128,000 in triage labor for every $315 in API costs—will catch many organizations off guard as they scale AI scanning across portfolios.

  • +1 Runtime reachability analysis will emerge as the critical differentiator, enabling teams to ignore non-exploitable findings and focus on vulnerabilities that attackers can actually reach.

▶️ Related Video (82% Match):

https://www.youtube.com/watch?v=7ee63G1q3hU

🎯Let’s Practice For Free:

🎓 Live Courses & Certifications:

Join Undercode Academy for Verified Certifications

🚀 Request a Custom Project:

Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

IT/Security Reporter URL:

Reported By: https://lnkd.in/p/eBc4ZTJz – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

💬 Whatsapp | 💬 Telegram

📢 Follow UndercodeTesting & Stay Tuned:

𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: