Listen to this Post
Introduction:
The Tor network hosts thousands of .onion services—many of which are legitimate, while others operate in the shadows of cybercrime, data breaches, and illegal marketplaces. For security researchers, law enforcement, and threat intelligence analysts, manually navigating this anonymous ecosystem is impractical and dangerous. TorBot—an open‑source Python‑based intelligence crawler—automates the discovery, enumeration, and mapping of Tor hidden services, transforming dark web reconnaissance from a tedious manual process into a structured, repeatable, and scalable OSINT operation.
Learning Objectives:
- Understand the architecture and capabilities of TorBot for dark web OSINT collection.
- Install, configure, and execute TorBot across Linux and Windows environments.
- Leverage TorBot’s crawling, link analysis, and visualization features to map .onion infrastructures.
- Apply security best practices to protect your identity and infrastructure when conducting dark web research.
- Extend TorBot’s functionality with custom scripts and integrate it into broader threat intelligence workflows.
You Should Know:
- What TorBot Is and Why It Matters for Cybersecurity
TorBot is an open‑source intelligence (OSINT) tool developed in Python, originally created by DedSecInside and now maintained under the OWASP umbrella. Its core purpose is to systematically crawl .onion domains, extract page titles, metadata, and link structures, and build an interactive tree graph of a target site’s architecture and its external connections. Unlike generic web crawlers, TorBot routes all traffic through the Tor network via SOCKS5 proxies, ensuring both the researcher’s anonymity and the ability to access hidden services that are unreachable over the clearnet.
For cybersecurity professionals, TorBot serves multiple use cases:
- Threat intelligence gathering – Monitor dark web forums, marketplaces, and criminal infrastructure for mentions of your organization, leaked credentials, or emerging threats.
- Vulnerability research – Map the attack surface of .onion services to identify misconfigurations, outdated software, or exposed sensitive information.
- Compliance and fraud detection – Detect brand impersonation, counterfeit goods, or illegal data trading involving your assets.
- Academic and law enforcement research – Collect structured data for analyzing dark web trends, criminal networks, and anonymization techniques.
The tool’s value lies in its automation: what would take a human analyst days or weeks to manually enumerate can be accomplished in hours with TorBot, while also producing visualizations that reveal hidden relationships between sites.
2. Installing TorBot on Linux and Windows
Prerequisites:
- Python 3.7 or higher
- Tor service running locally (or access to a remote Tor SOCKS5 proxy)
- Git (for cloning the repository)
- pip (Python package manager)
Linux (Ubuntu/Debian) Installation:
Update system packages sudo apt update && sudo apt upgrade -y Install Tor and essential dependencies sudo apt install tor python3 python3-pip git -y Start and enable Tor service sudo systemctl start tor sudo systemctl enable tor Verify Tor is running (SOCKS5 proxy on 127.0.0.1:9050) curl --socks5-hostname 127.0.0.1:9050 https://check.torproject.org/api/ip Clone the TorBot repository git clone https://github.com/DedSecInside/TorBot.git cd TorBot Install Python dependencies pip3 install -r requirements.txt
Windows Installation:
- Install Python – Download and install Python 3.7+ from python.org. Ensure “Add Python to PATH” is checked during installation.
- Install Tor – Download the Tor Expert Bundle from the Tor Project website. Extract it to a folder (e.g.,
C:\Tor). Run `tor.exe` to start the Tor service on127.0.0.1:9050. - Install Git – Download and install Git for Windows from git-scm.com.
4. Open Command Prompt or PowerShell and run:
git clone https://github.com/DedSecInside/TorBot.git cd TorBot pip install -r requirements.txt
Verification: After installation, run a quick test to confirm TorBot can communicate with the Tor proxy:
python3 torBot.py --help
If the help menu appears, the installation is successful. If you encounter proxy errors, ensure Tor is running and the SOCKS5 port (default 9050) is accessible.
3. Configuring TorBot for Anonymous Dark Web Crawling
TorBot relies on a properly configured Tor SOCKS5 proxy. By default, it expects the proxy at 127.0.0.1:9050. However, you can customize this and other settings via command‑line flags or by editing the configuration file.
Key Configuration Options:
| Flag | Description |
||-|
| `–proxy` | Specify a custom SOCKS5 proxy (e.g., `–proxy 127.0.0.1:9150` for Tor Browser’s proxy) |
| `–depth` | Set the crawl depth (number of links to follow from the seed URL). Default is 1. |
| `–output` | Save results to a file (JSON, CSV, or plain text) |
| `–visualize` | Generate an interactive tree graph of the crawled site structure |
| `–threads` | Number of concurrent threads for faster crawling (use with caution to avoid detection) |
| `–timeout` | Request timeout in seconds |
Example Configuration for Deep Crawling:
python3 torBot.py --url http://example.onion --depth 3 --threads 5 --visualize --output results.json
This command crawls `example.onion` to a depth of 3 levels, uses 5 threads, generates a visualization, and saves the results to results.json.
Security Best Practices for Configuration:
- Always verify Tor is running before starting TorBot. Use `curl –socks5-hostname 127.0.0.1:9050 https://check.torproject.org/api/ip` to confirm your IP is masked.
- Avoid high thread counts – Aggressive crawling can overload .onion services, potentially alerting administrators or causing denial‑of‑service conditions. Start with `–threads 1` and increase gradually.
- Use a dedicated research VM – Isolate your OSINT activities from your daily‑use machine to prevent accidental leakage of identifying information.
- Rotate Tor circuits – Restart Tor or use the `–1ew‑circuit` flag (if implemented) to change your exit node periodically, reducing the risk of correlation attacks.
- Running Your First TorBot Crawl and Analyzing Results
Step‑by‑Step Guide:
- Start Tor – Ensure the Tor service is active. On Linux:
sudo systemctl start tor. On Windows: run `tor.exe` from the extracted Tor folder. -
Identify a Target .onion URL – For testing, you can use known legitimate .onion services such as DuckDuckGo’s onion site (`https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/`). Never target illegal or non‑consensual sites without proper authorization.
3. Run a Basic Crawl:
python3 torBot.py --url https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ --depth 1
4. Interpret the Output:
- The console will display each discovered page title and its .onion address.
- Links found on each page are listed hierarchically.
- If `–visualize` is used, an HTML file (e.g.,
graph.html) is generated—open it in a browser to view an interactive tree map of the site’s structure.
5. Save and Export Data:
- Use `–output results.csv` to save structured data for further analysis in Excel, Splunk, or a SIEM.
- For advanced use, pipe the output to other OSINT tools like SpiderFoot or Maltego for correlation with clearnet intelligence.
Example Output (truncated):
[] Crawling: https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ [+] DuckDuckGo — Privacy, simplified. [+] Links found: - https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/about - https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/privacy - https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/terms
Troubleshooting Common Issues:
- “Connection refused” – Tor is not running or the proxy port is incorrect. Verify with
netstat -an | grep 9050. - “Timeout” – The .onion site may be slow or offline. Increase `–timeout` (e.g.,
--timeout 30). - “No module named ‘xxx’” – Missing Python dependencies. Run `pip install -r requirements.txt` again.
5. Advanced Techniques: API Security, Visualization, and Integration
TorBot is not just a standalone crawler—it can be embedded into larger threat intelligence pipelines.
API Security Considerations:
- If you expose TorBot’s functionality via an API (e.g., using Flask or FastAPI), never run it on a public‑facing server without strict authentication and rate limiting. Attackers could abuse your API to launch anonymized scans against targets, making you liable.
- Use API keys, JWT tokens, or mutual TLS to authenticate clients.
- Implement request logging and monitoring to detect abuse patterns.
Visualization for Threat Mapping:
TorBot’s `–visualize` flag generates an interactive D3.js‑based graph. This visualization helps analysts quickly identify:
– Central hubs (highly interconnected sites)
– Isolated services (potential hidden infrastructures)
– Unusual link patterns (possible command‑and‑control or data exfiltration paths)
To customize the visualization, you can modify the `visualize.py` script to change colors, node sizes, or add custom metadata.
Integration with SIEM and Threat Intelligence Platforms:
- Export TorBot results in JSON or CSV and ingest them into Splunk, Elasticsearch, or TheHive.
- Use `jq` to parse JSON output and feed it into MISP (Malware Information Sharing Platform) for automated threat sharing.
- Combine with Shodan or Censys data to correlate dark web findings with clearnet exposed assets.
Example: Parsing TorBot JSON with jq:
python3 torBot.py --url http://example.onion --depth 2 --output raw.json
cat raw.json | jq '.[] | {title: .title, url: .url, links: .links}' > parsed.json
- Cloud Hardening and Operational Security for OSINT Researchers
Running TorBot from a cloud instance (AWS, Azure, GCP) introduces unique risks:
– IP reputation – Cloud provider IP ranges are often blocked by .onion services or monitored by adversaries.
– Persistent identity – If your cloud instance is compromised, your entire OSINT operation could be exposed.
Hardening Checklist:
| Control | Implementation |
||-|
| Use ephemeral VMs | Launch a fresh instance for each campaign; destroy it after completion. |
| Route all traffic through Tor | Configure system‑wide SOCKS5 proxy or use `proxychains` to force all traffic through Tor. |
| Encrypt persistent storage | Use LUKS (Linux) or BitLocker (Windows) to protect stored results. |
| Implement strict firewall rules | Allow only outbound connections to Tor and inbound SSH from trusted IPs. |
| Enable comprehensive logging | Log all TorBot activities, system access, and network connections for forensic review. |
| Use a VPN before Tor | For additional anonymity, connect to a VPN first, then route through Tor (VPN → Tor → Target). |
Linux Command to Force All Traffic Through Tor using proxychains:
sudo apt install proxychains4 Edit /etc/proxychains4.conf and set: socks5 127.0.0.1 9050 proxychains4 python3 torBot.py --url http://example.onion
Windows Alternative: Use Proxifier or SocksCap to force TorBot.exe and all child processes through the Tor SOCKS5 proxy.
7. Extending TorBot: Custom Scripts and Automation
TorBot’s modular Python architecture allows you to write custom plugins or scripts to:
– Extract specific data types (e.g., email addresses, Bitcoin wallets, phone numbers) using regex.
– Perform sentiment analysis on forum posts.
– Detect changes in .onion content over time (diff‑based monitoring).
Example: Custom Script to Extract Email Addresses
import re
from torBot import TorBot
def extract_emails(text):
return re.findall(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[a-zA-Z]{2,}', text)
bot = TorBot()
bot.crawl('http://example.onion', depth=1)
for page in bot.pages:
emails = extract_emails(page.content)
if emails:
print(f"[+] Found emails on {page.url}: {emails}")
Automation with Cron (Linux) or Task Scheduler (Windows):
Schedule regular crawls to monitor changes in dark web sites. For example, a daily cron job:
0 6 cd /home/user/TorBot && python3 torBot.py --url http://target.onion --depth 2 --output /var/osint/daily_$(date +\%Y\%m\%d).json
Alerting: Combine with `curl` or webhooks to send notifications to Slack, Teams, or Discord when new threats are detected.
python3 torBot.py --url http://target.onion --depth 1 | grep -i "breach|leak|credential" && curl -X POST -H 'Content-type: application/json' --data '{"text":"Potential threat detected!"}' https://hooks.slack.com/services/...
What Undercode Say:
- Key Takeaway 1: TorBot transforms dark web OSINT from a manual, high‑risk activity into an automated, scalable process—but it must be used with strict operational security to avoid exposing the researcher’s identity.
- Key Takeaway 2: The true power of TorBot lies not in the crawl itself, but in the integration of its output with broader threat intelligence platforms, enabling correlation between dark web activities and clearnet attack surfaces.
Analysis: The resurgence of interest in tools like TorBot reflects a broader shift in cybersecurity: defensive teams are no longer waiting for breaches to occur—they are proactively hunting threats in the very places where attackers plan and execute their campaigns. However, this proactive stance comes with significant responsibility. Researchers must navigate legal, ethical, and technical minefields, from ensuring they have proper authorization to targeting only consenting or publicly accessible .onion services, to implementing robust anonymization techniques that prevent retaliation or legal exposure. TorBot, while powerful, is just one piece of the puzzle. Its effectiveness depends on the analyst’s ability to interpret the data, avoid false positives, and maintain a continuous feedback loop between dark web intelligence and on‑premise defenses. Organizations that successfully integrate TorBot into their threat hunting workflows will gain a critical advantage—early warning of emerging threats, brand abuse, and credential leaks—long before they manifest as incidents.
Prediction:
- +1 As regulatory frameworks around cyber threat intelligence sharing mature (e.g., GDPR, CCPA, and emerging dark web monitoring mandates), open‑source tools like TorBot will become standard components of corporate security stacks, driving demand for certified OSINT professionals.
- +1 The integration of AI and machine learning with TorBot’s crawling engine will enable automated threat classification, reducing false positives and allowing analysts to focus on high‑priority alerts.
- -1 Adversaries will increasingly deploy anti‑crawling techniques—such as dynamic content, CAPTCHAs, and rate limiting—specifically designed to thwart tools like TorBot, forcing continuous evolution of the crawler’s evasion capabilities.
- -1 The misuse of TorBot by malicious actors for reconnaissance against legitimate .onion services (or even clearnet targets via misconfigured proxies) will lead to increased scrutiny and potential legal restrictions on the distribution of such tools.
▶️ Related Video (78% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: 0xfrost Tor – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
