Listen to this Post
Introduction:
The lines between data gathering and intelligence analysis are blurring as Tom Vaillant open-sources a massive database of 7,500 OSINT tools alongside a structured `Skill.md` file designed to supercharge AI agents. This release, hosted on Hugging Face and GitHub, provides investigators, cybersecurity professionals, and journalists with a curated arsenal and a methodological framework that can be directly ingested by autonomous agents. By integrating with the upcoming OSINT Navigator (ETA March 25) and leveraging the Model Context Protocol (MCP), this initiative promises to transform how we conduct digital reconnaissance, corporate tracing, and threat intelligence.
Learning Objectives:
- Understand the structure and contents of the 7,500‑tool OSINT database and the `Skill.md` repository.
- Learn to deploy and query these resources using Linux, Python, and AI agent frameworks.
- Master practical OSINT techniques—from tool selection to “follow‑the‑money” investigations—with step‑by‑step commands and code.
You Should Know
- Navigating the OSINT Tool Database on Hugging Face
The Hugging Face dataset (tomvaillant/osint-tool-database) contains a CSV file listing 7,500+ tools, each tagged with categories (e.g., email, social media, domain, dark web), platform support, and a brief description. This is your starting point for discovering the right tool for any investigation.
Step‑by‑step guide:
1. Access the dataset
You can browse online at:
https://huggingface.co/datasets/tomvaillant/osint-tool-database`wget`:
<h2 style="color: yellow;">Or download it directly using
wget https://huggingface.co/datasets/tomvaillant/osint-tool-database/resolve/main/tools.csv
2. Explore with Python (pandas)
Install pandas and load the data:
pip install pandas python3
import pandas as pd
df = pd.read_csv('tools.csv')
print(df.head())
print(df['category'].value_counts()) see tool categories
3. Filter for specific needs
For example, to find all email‑related tools:
email_tools = df[df['category'].str.contains('email', case=False)]
print(email_tools[['name', 'url', 'description']])
Export the filtered list:
email_tools.to_csv('email_osint_tools.csv', index=False)
- Setting Up the OSINT Skills Repository for AI Agents
The `skills` repository (https://github.com/buriedsignals/skills`) provides a `Skill.md` file that defines three core commands:/osint,/investigate, and/follow-the-money`. This file can be ingested by AI agents (like OpenClaw or custom LangChain agents) to enable structured OSINT workflows.
Step‑by‑step guide:
1. Clone the repository
git clone https://github.com/buriedsignals/skills.git cd skills
2. Understand the Skill.md structure
Use `cat` or a text editor to review:
cat Skill.md
The file contains markdown with embedded JSON examples and natural language instructions. For instance, `/osint` routes queries to 150 specific tools with OPSEC notes.
3. Integrate with a Python‑based AI agent
A simple agent using LangChain can load the skill file as a prompt template. Example snippet:
from langchain import LLMChain, PromptTemplate
from langchain.llms import OpenAI
with open('Skill.md', 'r') as f:
skill_content = f.read()
prompt = PromptTemplate(
input_variables=["query"],
template=skill_content + "\n\nUser query: {query}\nAgent response:"
)
chain = LLMChain(llm=OpenAI(), prompt=prompt)
print(chain.run("Find all public records for a company in Panama"))
- Building an OSINT Lab with Kali Linux and Essential Tools
A dedicated environment ensures both power and OPSEC. Kali Linux offers a pre‑packaged OSINT suite, but you can also use the Hugging Face database to discover niche tools.
Step‑by‑step guide:
- Set up Kali Linux (or WSL with Kali)
– Download from https://www.kali.org/get-kali/ and install in a VM, or
– For Windows, enable WSL2 and install Kali from the Microsoft Store.
2. Update and install core OSINT tools
sudo apt update sudo apt install theharvester recon-ng maltego -y
- Use the Hugging Face dataset to install additional tools
Suppose you want all tools tagged “github”. Export the list and create an install script:github_tools = df[df['url'].str.contains('github.com', na=False)] with open('install_github_tools.sh', 'w') as f: for url in github_tools['url']: f.write(f"git clone {url}\n")
Then run the script (review first!):
chmod +x install_github_tools.sh ./install_github_tools.sh
4. Automating Investigations with the Navigator and MCP
The upcoming OSINT Navigator (March 25) will expose an MCP‑compatible API. This allows AI agents to query the Navigator for real‑time tool recommendations based on investigation context.
Step‑by‑step guide (simulated pre‑release):
1. Simulate an MCP query
Assuming the Navigator listens on `http://localhost:8080/mcp`, you can send a JSON request:
curl -X POST http://localhost:8080/mcp \
-H "Content-Type: application/json" \
-d '{"command": "recommend_tools", "context": "email header analysis"}'
2. Parse the response in Python
import requests
response = requests.post('http://localhost:8080/mcp', json={
'command': 'recommend_tools',
'context': 'email header analysis'
})
tools = response.json().get('tools', [])
for tool in tools:
print(f"{tool['name']}: {tool['url']}")
3. Feed the output into your investigation workflow
The agent can then automatically launch the recommended tools (e.g., using subprocess) or present them to the user.
5. Following the Money: Corporate and Asset Tracing
The `/follow-the-money` skill provides methodology for tracing corporate ownership and assets using open data sources like OpenCorporates, ICIJ Offshore Leaks, and sanctions lists.
Step‑by‑step guide:
1. Use OpenCorporates API
Get a free API key from https://opencorporates.com/api and search for a company:
curl "https://api.opencorporates.com/v0.4/companies/search?q=ACME&api_token=YOUR_KEY"
2. Parse with jq for clarity
curl -s "https://api.opencorporates.com/v0.4/companies/search?q=ACME&api_token=YOUR_KEY" | jq '.results.companies[] | {name: .company.name, jurisdiction: .company.jurisdiction_code}'
3. Cross‑reference with sanctions lists
Download the EU sanctions list (CSV) and grep for matches:
wget https://webgate.ec.europa.eu/fsd/fsf/public/files/csvFullSanctionsList/content?token=YOUR_TOKEN -O sanctions.csv grep -i "ACME" sanctions.csv
4. Automate with a Python script
Combine multiple sources and output a report. Example using `requests` and pandas:
import requests, pandas as pd ... (code to query OpenCorporates, sanctions, etc.) ...
6. Operational Security (OPSEC) for OSINT Agents
When deploying AI agents for OSINT, ensuring anonymity and avoiding detection is critical. The Skill.md file includes OPSEC notes, but you must implement them.
Step‑by‑step guide:
1. Route all traffic through Tor
Install Tor and configure proxy:
sudo apt install tor sudo systemctl start tor export HTTP_PROXY="socks5://127.0.0.1:9050" export HTTPS_PROXY="socks5://127.0.0.1:9050"
2. Verify your IP is hidden
curl ifconfig.me curl --socks5 127.0.0.1:9050 ifconfig.me should show Tor exit node
3. Use a VPN as fallback
If Tor is too slow, use a trusted VPN. Check for DNS leaks:
nmcli dev show | grep DNS or on Windows: ipconfig /all | findstr DNS
4. Isolate the investigation environment
Run your tools inside a Docker container with no persistent storage:
docker run --rm -it --network none kalilinux/kali-rolling
(Adjust network settings to use Tor proxy.)
- Integrating OSINT Tools with Python for Custom Workflows
Combine the tool database and skills to create a custom investigation assistant.
Step‑by‑step guide:
1. Load the tool database
import pandas as pd
df = pd.read_csv('tools.csv')
2. Create a simple recommender
Use keyword matching:
def recommend_tools(query, df):
mask = df['description'].str.contains(query, case=False, na=False) | df['category'].str.contains(query, case=False, na=False)
return df[bash][['name', 'url']].head(5)
print(recommend_tools('social media', df))
3. Integrate with the skills file
Parse `Skill.md` to extract step‑by‑step investigation methods for a given command. For instance, you can grep the `/investigate` section and feed it to an LLM to generate a checklist.
4. Run an automated investigation
Example: given a domain, the script first recommends tools, then runs them in sequence, collating output into a report.
What Undercode Say
- Key Takeaway 1: The open‑sourcing of 7,500 OSINT tools removes barriers to entry—anyone with basic technical skills can now access a curated, categorized arsenal for digital investigations. This levels the playing field for independent journalists and smaller security teams.
- Key Takeaway 2: By providing a machine‑readable
Skill.md, Vaillant enables AI agents to execute OSINT methodologies autonomously. This shifts the investigator’s role from manually juggling tools to supervising intelligent workflows, dramatically increasing efficiency. - Analysis: The combination of a comprehensive tool database and structured agent skills creates a powerful synergy. The upcoming Navigator with MCP will likely become a central hub for AI‑driven OSINT, allowing agents to dynamically discover and invoke the right tools based on context. However, this also raises concerns about misuse; OPSEC and ethical guidelines are more important than ever. The community must develop standards for responsible automation to prevent mass surveillance or privacy violations. This release is a milestone—it not only catalogs what exists but also provides the blueprint for how to use it intelligently.
Prediction
Within the next 12 months, we will witness a wave of autonomous investigator agents built on these open resources. The integration of MCP will enable interoperability between different OSINT platforms, leading to a “plug‑and‑play” ecosystem where agents can seamlessly combine data from multiple sources. Expect to see AI‑generated investigation reports, real‑time alerts on corporate changes, and even predictive analytics for emerging threats. As these tools become more accessible, regulatory bodies may step in to define acceptable use, but the cat is already out of the bag—OSINT is entering a new era of automation and scale.
▶️ Related Video (86% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Tomvaillant Open – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
