Listen to this Post
Introduction:
In the ever-escalating arms race of web application security, the gap between automated scans and manual penetration testing is often filled by costly enterprise tools. However, a new wave of innovation is emerging from academic projects that mimic real-world workflows. The development of “Scanly,” an automated web vulnerability scanner, represents a shift toward modular, behavior-based analysis that prioritizes accuracy and minimizes the false positives that plague traditional tools.
Learning Objectives:
- Understand the modular architecture of automated vulnerability scanners.
- Learn how to analyze HTTP responses to detect SQL Injection and Reflected XSS.
- Explore techniques for endpoint enumeration and risk classification.
- Gain practical knowledge on setting up a scanning environment using open-source alternatives.
You Should Know:
1. The Engine Behind Scanly: Modular Architecture Explained
Scanly’s strength lies in its decoupled, modular design. By separating the crawling, injection, detection, and reporting phases, the system allows for granular control and easier debugging. This architecture mimics professional tools like Burp Suite or OWASP ZAP but is tailored for automation.
What it does: It systematically discovers endpoints, tests them with payloads, analyzes the server’s response, and determines if a vulnerability exists based on behavioral anomalies.
How to replicate the logic conceptually:
While the source code for Scanly is proprietary to the team, you can simulate its workflow using a combination of tools.
Step-by-step guide (Linux Conceptual Simulation):
- Crawling & Enumeration: Use `gobuster` or `ffuf` to discover hidden directories and endpoints.
Enumerate directories on a target gobuster dir -u http://testphp.vulnweb.com -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt
- Parameter Discovery: Use `waybackurls` or `gau` to find historical parameters.
Fetch known URLs from the target echo "testphp.vulnweb.com" | gau
2. SQL Injection Detection via Response Analysis
The project highlights detecting SQL Injection using “response behavior analysis.” Instead of just looking for “mysql_error” in the text, a robust scanner analyzes the differences between a baseline request and a malicious request.
Step‑by‑step guide:
You can test this concept manually or with a simple Python script.
1. Identify a test parameter: Find a URL like http://testphp.vulnweb.com/artists.php?artist=1`.‘ OR ‘1’=’1`.
2. Send a Baseline Request: Note the response length and content (e.g., "Artist: John Doe").
3. Inject a SQL Payload: Append a payload like
Using cURL to test for error-based SQLi curl "http://testphp.vulnweb.com/artists.php?artist=1'+OR+'1'%3D'1"
4. Analyze the behavior:
- Error-based: Look for database errors (MySQL syntax, ODBC drivers).
- Boolean-based: If the page content changes drastically (disappearance of text) compared to the baseline, it indicates a potential vulnerability.
3. Detecting Reflected XSS with Comparison Techniques
Reflected XSS is often missed by scanners that simply inject a payload and look for the exact string. Scanly’s method involves “response comparison” to ensure the injected script is reflected in the same unencoded context.
Step‑by‑step guide (Windows/Linux):
- Identify an input vector: A search bar, e.g., `http://testphp.vulnweb.com/search.php?test=query`.
- Inject a unique string: Use a value unlikely to appear naturally, like
ScanlyTest123.Linux curl "http://testphp.vulnweb.com/search.php?test=ScanlyTest123"
- Check for reflection: If the response body contains “ScanlyTest123,” the input is reflected.
- Escalate with a payload: Now inject `
XSS
` or
.URL Encoded payload: %3Ch1%3EXSS%3C%2Fh1%3E curl "http://testphp.vulnweb.com/search.php?test=%3Ch1%3EXSS%3C%2Fh1%3E"
- Context Analysis: If the HTML source shows `
XSS
` as raw HTML (not encoded as
<h1>), the application is vulnerable.
4. Implementing HTTP Request Injection & Response Analysis
The injection layer is responsible for fuzzing parameters with various payloads. This requires handling concurrent requests without overwhelming the server.
Python Snippet for Basic Injection Logic:
import requests
from urllib.parse import quote
Payload list for SQLi
payloads = ["'", "' OR '1'='1", "' UNION SELECT null--"]
target = "http://testphp.vulnweb.com/artists.php"
param = "artist"
base_url = f"{target}?{param}="
for payload in payloads:
encoded_payload = quote(payload)
full_url = base_url + encoded_payload
response = requests.get(full_url)
Response Analysis: Check for DB errors
if "mysql" in response.text.lower() or "sql" in response.text.lower():
print(f"[!] Potential SQLi with payload: {payload}")
else:
print(f"[-] No obvious error with payload: {payload}")
5. Structured Reporting & Risk Classification
A scanner is only as good as its report. Scanly classifies findings based on CVSS (Common Vulnerability Scoring System) vectors or custom business logic.
How to structure findings:
Create a JSON output that includes:
- Vulnerability: SQL Injection
- Risk: High (CVSS: 9.8)
- Endpoint: /artists.php
- Parameter: artist
- Payload: `’ OR ‘1’=’1`
– Evidence: “You have an error in your SQL syntax”
Linux Command to parse logs:
Use `jq` to filter high-risk vulnerabilities from a JSON report.
cat scan_results.json | jq '.[] | select(.risk == "High")'
6. Minimizing False Positives: The Logic Layer
The post emphasizes minimizing false positives. This is achieved by setting confidence thresholds. For example, if an error message contains “mysql,” it might be a false positive if the application is designed to show database errors to users. Scanly likely compares the response against a baseline of “normal” server errors.
Mitigation Strategy:
- Whitelist Responses: If the page contains “Artist:” in a SQLi test, but after injection it does not, flag it. If it contains “error” but also “Artist,” it might be a benign server warning, not an exploitable injection.
7. Cloud & API Security Considerations (Extrapolation)
While Scanly focuses on web apps, its modular structure is ideal for API security testing. By modifying the “HTTP Request Injection” layer to handle JSON/XML bodies, the scanner could test REST APIs for NoSQL injection or mass assignment.
Windows Command to test API endpoint:
Test a login API for mass assignment
curl -X POST https://target.com/api/login -H "Content-Type: application/json" -d '{\"user\":\"admin\",\"pass\":\"123\",\"isAdmin\":true}'
What Undercode Say:
- Key Takeaway 1: The future of security tools lies in “behavioral analysis” rather than signature-based detection, drastically reducing false positives.
- Key Takeaway 2: Modular design is critical; decoupling the scanning engine allows security teams to replace or upgrade individual components (like the crawler) without rebuilding the entire tool.
The “Scanly” project underscores a vital truth: effective security automation must understand the context of an application. By focusing on how a server behaves when injected with malicious data, rather than just scanning for known strings, even a graduation project can rival professional tools in accuracy. This approach democratizes security testing, proving that innovation often starts in the classroom.
Prediction:
We will see a surge in AI-integrated scanners that learn application behavior over time, dynamically adjusting payloads based on the server’s response patterns, moving us closer to fully autonomous penetration testing agents that require zero human intervention for common vulnerability classes.
▶️ Related Video (90% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Rabeiy Badry – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
