Listen to this Post
Introduction:
Modern penetration testing and red teaming hinge on one critical phase: reconnaissance. Relying on a single tool creates blind spots, but combining multiple intelligence sources—from internet-wide scanners to code repositories and email intelligence—enables analysts to build a complete, actionable picture of any target. This article transforms a curated list of over 20 specialized search engines into a structured, hands-on methodology for infrastructure mapping, threat intelligence correlation, and attack surface validation.
Learning Objectives:
- Integrate Shodan, Censys, and Netlas for comprehensive infrastructure discovery
- Automate subdomain enumeration using crt.sh, Google Dorks, and PublicWWW
- Correlate threat intelligence with GreyNoise, Pulsedive, and Vulners for exposure validation
- Execute OSINT workflows on both Linux and Windows using command-line and API-based techniques
You Should Know:
1. Infrastructure Mapping with Shodan, Censys, and Netlas
These platforms index internet-facing devices and services. Use them to discover open ports, vulnerable banners, and unexpected exposures.
Step‑by‑step guide (Linux):
Shodan CLI installation and basic query pip install shodan shodan init YOUR_API_KEY shodan search 'org:"Target Corp" port:22' --limit 10 --fields ip_str,port,org Censys search via API (requires Python) pip install censys censys search 'services.port=443 and services.service_name="nginx"' --limit 5 Netlas query for JIRA instances curl -X GET "https://app.netlas.io/api/search/?q=product:jira&fields=host" -H "X-API-Key: YOUR_KEY"
Windows PowerShell equivalent:
Shodan using Invoke-RestMethod
$body = @{key="YOUR_API_KEY"; query="port:3389 country:DE"} | ConvertTo-Json
Invoke-RestMethod -Uri "https://api.shodan.io/shodan/host/search?key=YOUR_KEY&query=port:3389" -Method Get
What this does: Queries each engine’s API for hosts matching specific organizational fingerprints or service banners. Combine results into a unified CSV for attack surface visualization.
- Subdomain & Asset Discovery with crt.sh + Google Dorks + PublicWWW
crt.sh pulls certificate transparency logs; Google Dorks finds indexed pages; PublicWWW searches HTML/CSS source code across millions of websites.
Step‑by‑step guide (Linux):
Extract all subdomains from crt.sh for a domain curl -s "https://crt.sh/?q=%25.example.com&output=json" | jq -r '.[].name_value' | sort -u Google Dorks via command line (use googler tool) googler -1 50 "site:example.com intitle:index of" --json PublicWWW search - search for specific CSS class or JS variable curl -X POST "https://publicwww.com/api/search/" -d "q=bootstrap+cdn&format=json&page=1"
Windows (using PowerShell and curl):
crt.sh with Invoke-WebRequest $cert = Invoke-WebRequest -Uri "https://crt.sh/?q=%25.example.com&output=json" $cert.Content | ConvertFrom-Json | Select-Object -ExpandProperty name_value -Unique
Pro tip: Feed discovered subdomains into FullHunt or SecurityTrails for historical DNS records and takeovers.
3. Threat Intelligence Correlation: GreyNoise, Pulsedive, Vulners
GreyNoise differentiates opportunistic scanners from targeted threats; Pulsedive enriches indicators; Vulners maps CVEs to exploits.
Step‑by‑step guide (API automation):
!/usr/bin/env python3
import requests
GreyNoise: check if an IP is a benign scanner
gn_key = "YOUR_GN_KEY"
ip = "45.155.205.233"
r = requests.get(f"https://api.greynoise.io/v3/community/{ip}", headers={"key": gn_key})
print("GreyNoise:", r.json().get("classification"))
Pulsedive: get indicator info
pid = "https://pulsedive.com/api/info.php?indicator=8.8.8.8&pretty=1"
print(requests.get(pid).json())
Vulners: search for exploits by CVE
vuln = "https://vulners.com/api/v3/search/lucene/?query=CVE-2024-6387&apiKey=YOUR_VULNERS_KEY"
print(requests.get(vuln).json())
Run this script daily to correlate IPs from your firewall logs. If GreyNoise says “malicious” and Vulners shows a recent exploit, prioritize blocking.
- Attack Surface Validation with URLScan, FOFA, and ZoomEye
These services capture screenshots, DOM structures, and HTTP responses for any URL. Use them to verify live assets before exploitation.
Step‑by‑step guide (no coding required for basic use, but automation helps):
Submit a URL to URLScan and retrieve result
curl -X POST "https://urlscan.io/api/v1/scan/" -H "Content-Type: application/json" -d '{"url": "https://target.com", "visibility": "public"}'
Save the response, extract "uuid", then:
curl "https://urlscan.io/api/v1/result/YOUR_UUID/" | jq '.page.domain, .lists.urls'
FOFA search (requires API key)
curl "https://fofa.info/api/v1/search/all?key=YOUR_KEY&qbase64=dGl0bGU9IlB5dGhvbiI%3D&size=5"
ZoomEye host search
curl "https://api.zoomeye.org/host/search?query=port:22" -H "Authorization: JWT YOUR_TOKEN"
Windows users can use the same curl commands if curl is installed (Windows 10+ includes it). For persistent scanning, schedule these as PowerShell background jobs.
- OSINT & Data Exposure: Intelligence X, LeakIX, Hunter
Intelligence X archives deleted or dark web content; LeakIX maps breach exposures; Hunter finds email patterns.
Step‑by‑step guide (manual + automation):
Hunter: discover email addresses for a domain
curl "https://api.hunter.io/v2/domain-search?domain=example.com&api_key=YOUR_KEY" | jq '.data.emails[].value'
Intelligence X terminal search (requires login token)
curl -X POST "https://2.intelx.io/phonebook/search" -H "x-key: YOUR_KEY" -d '{"term":"example.com","maxresults":10}'
Check search status
curl "https://2.intelx.io/search/result?uuid=RESULT_ID" -H "x-key: YOUR_KEY"
LeakIX - query by IP or domain
curl "https://leakix.net/search?q=example.com" -H "Accept: application/json" | jq '.results[] | {host: .host, leak: .leak_type}'
What this does: Maps exposed credentials, API keys, and email addresses belonging to your target. Combine with haveibeenpwned API for breach correlation.
6. Code & Web Intelligence: grep.app, Searchcode, PublicWWW
Search code repositories and web pages for accidentally committed secrets, API keys, or internal URLs.
Step‑by‑step guide (regex-focused):
grep.app - search for AWS keys across public repos
curl "https://grep.app/api/search?q=AKIA[0-9A-Z]{16}®exp=true&files=match" | jq '.hits.hits[].code'
Searchcode - find .env files
curl "https://searchcode.com/api/codesearch_I/?q=.env&p=1&per_page=10"
PublicWWW to find pages containing a specific Google Analytics ID
curl "https://publicwww.com/api/search/?q=UA-123456-1&format=json"
Security warning: Do not download or use any credentials you discover without explicit permission. These techniques are for authorized red team exercises only.
- Email & Identity Reconnaissance (Hunter + domain permutations)
Beyond basic domain search, use email permutation logic to guess employee addresses.
Step‑by‑step guide (Python script):
import requests, itertools
domain = "target.com"
api_key = "HUNTER_API_KEY"
patterns = ["first.last", "flast", "firstl", "first.last+alias"]
Get verified emails from Hunter
r = requests.get(f"https://api.hunter.io/v2/domain-search?domain={domain}&api_key={api_key}")
verified = [e['value'] for e in r.json()['data']['emails'] if e['verified']]
Generate permutations from common names (mock list)
first_names = ["john","jane"]
last_names = ["smith","doe"]
for first,last in itertools.product(first_names, last_names):
for p in patterns:
print(p.replace("first",first).replace("last",last) + "@" + domain)
Export results and test via verified SMTP (with permission) or feed into LinkedIn OSINT tools.
What Undercode Say:
- Master the recon cycle: Infrastructure mapping + code search + threat intel yields more vulnerabilities than blind scanning.
- Automation is key – combine APIs from Shodan, GreyNoise, and Hunter into a single Python dashboard.
- Windows pentesters often ignore CLI tools; curl, jq, and PowerShell are equally powerful for OSINT.
-
Analysis (10 lines): The post emphasizes that beginners fixate on one tool while experts weave multiple data sources. This is critical because modern attack surfaces are ephemeral – containers, serverless functions, and CDNs appear and disappear. Using only Shodan misses code leaks on grep.app; relying solely on Hunter ignores exposure on LeakIX. The pro tip combinations (Shodan+Censys+Netlas for infrastructure, crt.sh+Google Dorks+1ublicWWW for assets) are battle-tested by red teams. However, the list lacks API rate-limiting handling and legal disclaimers – always ensure you have permission before scanning third-party assets. Incorporating these tools into a CI/CD pipeline for continuous monitoring gives defenders an edge. For offensive security, mastering search operators and automation scripts reduces manual effort by 80%. The future lies in agentic OSINT where LLMs query these engines autonomously and correlate findings. Finally, note that many of these services require paid API keys for full results; free tiers suffice for learning but not for large-scale campaigns.
Prediction:
+1 Open-source integration of these 20+ engines into unified reconnaissance frameworks (e.g., ReconFTW, Sn1per) will become standard by 2027, reducing human effort by 70%.
-1 Attackers will increasingly abuse the same OSINT platforms to automate victim profiling, leading to tighter API rate limiting and authentication requirements.
+1 Cloud providers will embed lightweight Shodan/Censys equivalents into their native security hubs, making infrastructure discovery accessible to junior engineers.
-1 Privacy regulations (GDPR, CCPA) may restrict the use of public search engines for security research without explicit consent, driving more threat intelligence underground.
+1 AI-powered correlation tools (e.g., using vector databases) will automatically map exposed subdomains to known exploits from Vulners, enabling real-time patch prioritization.
▶️ Related Video (80% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Vyankatesh Shinde – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
