AI-Powered Threat Intel Aggregator: Automate Dark Web Monitoring & IOC Extraction in 2 Minutes + Video

By /

Listen to this Post

Featured Image

Introduction:

Manual threat intelligence gathering from scattered RSS feeds, breach forums, and .onion sites is error-prone and unsustainable. “Threat Intel Nom Nom” leverages AI to automate aggregation, keyword-based alerting, and automatic extraction of indicators of compromise (IOCs) and CVEs – turning a morning hamster wheel into a one‑command Docker deployment.

Learning Objectives:

  • Deploy Threat Intel Nom Nom using Docker Compose with persistent storage and Tor proxy integration
  • Configure keyword alerting with regex and custom feed sources (including dark web .onion sites)
  • Automate IOC extraction (IPs, domains, hashes, CVEs) and route notifications to Discord, email, or webhook endpoints

You Should Know

1. One‑Command Deployment with Docker Compose

Threat Intel Nom Nom bundles a React frontend, FastAPI backend, PostgreSQL, Celery workers, Redis, and a Tor proxy. The official `docker-compose.yml` pulls all images and sets up networking.

Step‑by‑step guide:

  1. Install Docker and Docker Compose on your system.

– Linux (Ubuntu/Debian): 

sudo apt update && sudo apt install docker.io docker-compose -y
sudo systemctl enable --now docker

– Windows (WSL2 or Docker Desktop): Enable WSL2 backend, then install Docker Desktop.
2. Clone the repository (replace `` with the actual repo – extract from the LinkedIn short link): 

git clone https://github.com/<user>/threat-intel-nom-nom.git
cd threat-intel-nom-nom

3. Launch the stack:

docker-compose up -d

4. Verify services:

docker-compose ps

Expected: react, api, db, redis, celery, `tor-proxy` all “Up”.
5. Access the dashboard at `http://localhost:3000`. Default credentials (if any) are in the `.env.example` file.

What this does: The Compose file creates an isolated environment where the Tor proxy routes .onion requests, Celery handles periodic feed scraping, and PostgreSQL stores extracted IOCs.

2. Customising Threat Feeds (RSS, Websites, APIs, .onion)

The tool ships with 24 pre‑configured feeds (ransomware leak sites, security blogs, breach databases). You can add or remove feeds via the UI or a JSON config file.

Step‑by‑step guide:

  1. In the dashboard, navigate to Feeds → Add New.
  2. Choose feed type: RSS, HTML, API, or Onion.
  3. For an .onion feed, ensure the Tor proxy is running:

– Test Tor connectivity: 

docker exec -it tor-proxy curl --socks5-hostname localhost:9050 http://check.torproject.org/api/ip

Returns `{“IsTor”:true}` if working.

  1. Add a dark web feed (example – a breach site):

– URL: `http://breachsite123.onion/feed`
– Interval: `3600` seconds
– Keyword filters: `”ransomware”,”leak”,”credentials”`
5. Save – the Celery worker will start scraping.

Linux/Windows tip: To bulk import feeds, edit `data/feeds.json` inside the mounted volume. Use `jq` for validation: 

cat data/feeds.json | jq '.feeds[] | {url, type}'

3. Keyword Alerting with Regex & IOC Auto‑Extraction

The core AI logic matches incoming content against user‑defined keywords (supports regex). Matches are parsed for IOCs using pre‑built patterns.

Step‑by‑step guide:

1. Go to Alerts → Create Rule.

2. Example rule for CVE detection:

  • Name: `Critical CVEs`
  • Keywords: `\bCVE-\d{4}-\d{4,7}\b` (regex)
  • Case‑sensitive: No

3. The system automatically extracts:

  • IPv4: `\b(?:\d{1,3}\.){3}\d{1,3}\b`
  • Domains: `\b([a-z0-9]+(-[a-z0-9]+)\.)+[a-z]{2,}\b`
  • MD5/SHA1/SHA256 hashes: `\b[a-fA-F0-9]{32,64}\b`
  1. To test a feed manually, use the Test button – it shows matched content and extracted IOCs.
  2. All extracted IOCs are stored in PostgreSQL. Query them: 
    docker exec -it db psql -U nomnom -d threat_intel -c "SELECT  FROM iocs LIMIT 10;"

Pro tip: Combine keywords with regex lookaheads to filter context (e.g., "password.leak"). Avoid overly broad patterns to reduce false positives.

4. Notification Channels: Discord, Email, Webhook

Never miss an alert. Threat Intel Nom Nom supports multiple output channels. Webhooks are ideal for SIEM integration.

Step‑by‑step guide – Discord webhook:

  1. In Discord, go to Server Settings → Integrations → Create Webhook. Copy the URL.
  2. In the tool, navigate to Settings → Notifications → Add Channel.
  3. Select Discord, paste the webhook URL, and set alert severity threshold (e.g., High).

4. For email (SMTP), configure environment variables:

SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
[email protected]
SMTP_PASS=app_password

Restart the API container: `docker-compose restart api`.

5. Webhook (generic JSON POST):

  • URL: `https://your-siem.com/ingest`
  • Headers: `Content-Type: application/json`
  • The payload includes { "title", "description", "iocs": [...] }.

Security note: Never commit webhook URLs or SMTP credentials to version control. Use Docker secrets or a `.env` file excluded via .gitignore.

  1. Hardening the Tor Proxy & Dark Web Monitoring

Accessing .onion sites requires caution – misconfigured Tor can leak your IP or expose internal services.

Step‑by‑step guide for secure configuration:

  1. The built‑in Tor proxy uses default config. Edit `tor/torrc` to enforce: 
    SocksPort 0.0.0.0:9050
SocksPolicy accept 172.16.0.0/12  allow only Docker internal network
ExitNodes {us}  restrict exit country (optional)
StrictNodes 1

2. Restart Tor: `docker-compose restart tor-proxy`.

3. To verify no DNS leaks, run:

docker exec tor-proxy curl --socks5-hostname localhost:9050 https://check.torproject.org/api/ip

4. For production, isolate the Tor container on a dedicated bridge network and limit its outbound firewall rules (allow only port 9050).
5. Windows equivalent: Use Docker Desktop’s resource limits – assign only 1 CPU core to Tor to prevent resource exhaustion.

Warning: Monitoring .onion sites may be illegal in some jurisdictions. Ensure you have authorisation and operate within your organisation’s security policies.

6. API Security & Cloud Hardening

The FastAPI backend (port 8000) should never be exposed directly to the internet without authentication and rate limiting.

Step‑by‑step hardening:

1. Generate an API key for external tools:

docker exec -it api python -c "import secrets; print(secrets.token_urlsafe(32))"

2. Add the key to `API_KEYS` environment variable (comma‑separated).

3. Deploy behind a reverse proxy (nginx/traefik) with:

  • TLS 1.3 only
  • Rate limiting: `limit_req zone=api burst=5`
  • IP whitelisting (if possible)

4. For cloud (AWS, Azure, GCP):

  • Run the Docker stack on a private subnet.
  • Use a load balancer with WAF rules to block SQLi and path traversal.
  • Enable audit logging for all API calls: 
    docker-compose logs api | grep "GET /api/feeds"

5. Periodic vulnerability scanning:

docker scan threat-intel-nom-nom_api:latest

Linux command to monitor container resource usage:

docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

7. Extending with TAXII Server Integration

The LinkedIn comment suggested turning the tool into a TAXII server – a standard for sharing threat intelligence. While not native, you can easily forward extracted IOCs to a TAXII 2.1 endpoint.

Step‑by‑step guide (post‑extraction workflow):

  1. Deploy an open‑source TAXII server (e.g., `cTI` or OpenTAXII).
  2. In Threat Intel Nom Nom, create a webhook notification channel that points to your TAXII server’s `/collections/{id}/objects/` endpoint.
  3. The webhook payload must be transformed to STIX 2.1 format. Use a small Python script as middleware: 
    webhook_transformer.py
from flask import Flask, request
import requests, json
app = Flask(<strong>name</strong>)
@app.route('/transform', methods=['POST'])
def transform():
data = request.json
stix_bundle = {
"type": "bundle",
"objects": [{"type": "indicator", "pattern": f"[ipv4-addr:value = '{ioc}']" for ioc in data['iocs']}]
}
requests.post('http://taxii-server:9000/api/collections/1/objects/', json=stix_bundle)
return "OK", 200
  4. Run the transformer as a microservice. This bridges ad‑hoc alerts into a structured TAXII feed for sharing with MISP, TheHive, or other SOAR platforms.

What Undercode Say

  • Automation is mandatory, not optional – Manually triaging dozens of threat feeds wastes analyst cycles that could be spent on active hunting. Threat Intel Nom Nom proves that a weekend project with AI can replace a full‑time human task.
  • Dark web monitoring is now accessible – By bundling a Tor proxy and pre‑configured .onion scrapers, the tool lowers the barrier for small teams to detect leaks and ransomware chatter without expensive commercial solutions.

Analysis: The true innovation here isn’t the AI – it’s the integration pattern. Combining Docker, Celery, regex‑based IOC extraction, and pluggable notifications creates a blueprint for security automation. However, organisations must harden the deployment (especially Tor exit policies and API authentication) before production use. The tool also highlights a gap: most threat intel platforms still require manual feed curation. Future iterations could include ML classification to prioritise alerts by severity and relevance. Overall, this is a practical example of “shift‑left security” – giving engineers the ability to build their own intelligence pipelines without vendor lock‑in.

Prediction

Within 12–18 months, open‑source, AI‑driven threat aggregators like Threat Intel Nom Nom will become the standard for mid‑sized SOCs, replacing expensive legacy SIEM add‑ons. We’ll see a rise in “bring your own intel” architectures where companies customise feed sources, deploy edge‑based Tor proxies, and pipe extracted IOCs directly into SOAR playbooks. The downside: attackers will also use similar tools to monitor their own leak sites and adjust tactics faster. The arms race will shift from data collection to real‑time correlation and automated response – making AI not just a scraper, but a decision engine.

▶️ Related Video (80% Match):

🎯Let’s Practice For Free:

IT/Security Reporter URL:

Reported By: Daniel Scheidt – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

💬 Whatsapp | 💬 Telegram

📢 Follow UndercodeTesting & Stay Tuned:

𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: