Listen to this Post
Introduction:
The web remains humanity’s largest open repository, but AI agents have historically paid tolls to access it via third‑party search and rendering APIs. TinyFish changes this paradigm by offering free, production‑grade Web Search and Fetch endpoints that return structured JSON and clean markdown – eliminating the need to manage a custom browser fleet or pay per query. This democratisation allows developers, security researchers, and red teams to build autonomous agents that navigate JavaScript‑heavy sites, bypass anti‑bot measures, and extract intelligence at scale without recurring costs.
Learning Objectives:
- Obtain and configure a TinyFish API key for programmatic web search and headless browsing.
- Execute search queries and fetch rendered pages with markdown output using command‑line tools and Python.
- Integrate free web access into AI agent frameworks ( Code, OpenClaw, Cursor) for enhanced contextual retrieval.
- Apply security best practices for API key storage, rate limit management, and hardened agent deployment.
You Should Know:
- Acquiring Your Free API Key and Making the First Request
TinyFish provides a no‑credit‑card signup with generous rate limits. Visit their API portal (https://lnkd.in/gR4GQrFx) to generate a key. Once obtained, test both endpoints using `curl` – no agent framework required.
Step‑by‑step guide:
- Navigate to the signup link, create an account, and copy your API key (e.g.,
tf_abc123def). - Export the key as an environment variable to avoid hardcoding:
– Linux/macOS: `export TINYFISH_API_KEY=”your_key_here”`
– Windows (Command Prompt): `set TINYFISH_API_KEY=your_key_here`
– Windows (PowerShell): `$env:TINYFISH_API_KEY=”your_key_here”`
3. Perform a web search query:
curl -X GET "https://api.tinyfish.io/v1/search?q=OWASP+Top+10+2025" \ -H "Authorization: Bearer $TINYFISH_API_KEY"
The response returns structured JSON with title, URL, snippet, and metadata – ideal for agent parsing.
4. Fetch a full webpage as clean markdown:
curl -X POST "https://api.tinyfish.io/v1/fetch" \
-H "Authorization: Bearer $TINYFISH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/spa-app"}'
This renders JavaScript, resolves single‑page applications, bypasses basic anti‑bot, and outputs only the textual content in markdown format – reducing token consumption by up to 80% compared to raw HTML.
- Structured JSON Search – Building Automated Intelligence Pipelines
Unlike traditional search APIs that return HTML snippets, TinyFish returns a JSON schema optimised for agent retrieval. You can pipe results directly into LLM context or enrichment scripts.
Using `jq` to extract links and summaries (Linux):
curl -s -X GET "https://api.tinyfish.io/v1/search?q=CVE-2025+exploit" \
-H "Authorization: Bearer $TINYFISH_API_KEY" | jq '.results[] | {title, url}'
Python example for persistent monitoring:
import os, requests
API_KEY = os.getenv("TINYFISH_API_KEY")
headers = {"Authorization": f"Bearer {API_KEY}"}
def search_threat_intel(keyword):
resp = requests.get(f"https://api.tinyfish.io/v1/search?q={keyword}", headers=headers)
return resp.json()["results"]
Collect emerging ransomware indicators
for result in search_threat_intel("new ransomware group 2026"):
print(f"[+] {result['title']} -> {result['url']}")
This allows security analysts to build free, cron‑driven threat intelligence feeds without expensive commercial APIs.
3. Rendering JavaScript and Anti‑Bot Pages with Fetch
Many modern websites (login portals, dashboards, cloud consoles) rely on client‑side rendering and bot detection. TinyFish’s custom Chromium fleet executes all JavaScript, waits for network idle, and returns a distilled markdown version – perfect for scraping dynamic content that would otherwise require a full browser automation stack (Selenium, Puppeteer).
Tutorial: Extract a Cloud Hardening Guide from a React‑based Documentation Site
1. Identify a documentation page that loads content via AJAX (e.g., AWS EKS best practices).
2. Use the fetch endpoint with the `?wait_for=networkidle2` parameter:
curl -X POST "https://api.tinyfish.io/v1/fetch" \
-H "Authorization: Bearer $TINYFISH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://docs.example.com/security/iam-hardening", "wait_for": "networkidle2"}'
3. Save the markdown output to a file and feed it into an LLM for summarisation or compliance checking.
Windows PowerShell equivalent:
$body = @{url="https://docs.example.com/security/iam-hardening"; wait_for="networkidle2"} | ConvertTo-Json
Invoke-RestMethod -Uri "https://api.tinyfish.io/v1/fetch" -Method Post -Headers @{Authorization="Bearer $env:TINYFISH_API_KEY"} -Body $body -ContentType "application/json"
- Integrating with Agent Frameworks ( Code, OpenClaw, Cursor)
The post highlights compatibility with leading agent ecosystems. Free web access transforms these tools from static assistants into dynamic researchers.
For Code / Cursor – add a custom MCP (Model Context Protocol) server:
// mcp-tinyfish-server.js
import { MCPServer } from "@modelcontextprotocol/sdk";
const server = new MCPServer({ name: "tinyfish-web" });
server.addTool("web_search", async (query) => {
const res = await fetch(`https://api.tinyfish.io/v1/search?q=${query}`, {
headers: { Authorization: `Bearer ${process.env.TINYFISH_API_KEY}` }
});
return res.json();
});
server.start();
For OpenClaw (autonomous browser agent) – configure web access as a free skill:
openclaw_config.yaml
skills:
- name: tinyfish_fetch
command: curl -X POST https://api.tinyfish.io/v1/fetch -H "Authorization: Bearer $TINYFISH_API_KEY" -d '{"url": "$URL"}'
output_format: markdown
Once configured, agents can autonomously search for recent CVEs, fetch patch notes, and incorporate live data into decision‑making – all without incurring per‑request fees.
5. Rate Limits, Token Optimisation, and Cost Avoidance
TinyFish’s free tier includes “generous rate limits” – typically 100–500 requests per minute depending on endpoint. To stay within bounds and maximise efficiency:
Implement exponential backoff in Python:
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def fetch_safe(url):
return requests.post("https://api.tinyfish.io/v1/fetch", headers=headers, json={"url": url})
Reduce token waste by fetching only relevant sections:
The markdown output often includes navigation cruft. Use a summarisation LLM call with a low `max_tokens` to extract only the core paragraphs before feeding into a larger agent context.
Monitor usage via response headers:
curl -I -X GET "https://api.tinyfish.io/v1/search?q=test" -H "Authorization: Bearer $TINYFISH_API_KEY" Look for: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset
- Building a Fully Autonomous Web Research Agent (Python Tutorial)
Combine search + fetch to create an agent that answers natural language questions by browsing the live web.
Step 1 – Define the agent loop:
import os, requests
from openai import OpenAI or any LLM API
TF_KEY = os.getenv("TINYFISH_API_KEY")
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
def tiny_search(q):
r = requests.get(f"https://api.tinyfish.io/v1/search?q={q}", headers={"Authorization": f"Bearer {TF_KEY}"})
return r.json()["results"][:3] top 3 results
def tiny_fetch(url):
r = requests.post("https://api.tinyfish.io/v1/fetch", headers={"Authorization": f"Bearer {TF_KEY}"}, json={"url": url})
return r.json()["markdown"]
def agent(query):
1. Search
results = tiny_search(query)
2. Fetch each result's content
contexts = [tiny_fetch(r["url"]) for r in results]
3. LLM synthesis
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": f"Answer based on:\n{chr(10).join(contexts)}\n\nQuestion: {query}"}]
)
return response.choices[bash].message.content
print(agent("What are the latest AI supply chain attacks?"))
Step 2 – Deploy securely on a cloud VM (AWS EC2 / Azure VM):
– Store API keys in cloud secrets manager (AWS Secrets Manager or Azure Key Vault).
– Run the agent inside a container with read‑only filesystem to prevent key exfiltration.
– Set up network ACLs to allow outbound HTTPS only to `api.tinyfish.io` and your LLM provider.
- Security Hardening for API Key Usage and Agent Pipelines
Free APIs attract abuse; protect your key and your agent’s integrity.
Linux commands to lock down environment:
Create a dedicated service user sudo useradd -r -s /bin/false tinyfish_agent Store key in a root‑only file echo "TINYFISH_API_KEY=your_key" | sudo tee /etc/tinyfish/key.env sudo chmod 600 /etc/tinyfish/key.env sudo chown root:root /etc/tinyfish/key.env
Windows (PowerShell) – restrict key to service account:
$cred = Get-Credential Create a managed service account $env:TINYFISH_API_KEY = "your_key" Use PS Session configuration to limit visibility
Mitigation against prompt injection:
When an agent fetches external web pages, an attacker could embed malicious instructions in the retrieved markdown. Always sanitise fetched content by stripping markdown links and running an allow‑list filter before passing to the LLM.
import re def sanitize_markdown(md): Remove images and scripts md = re.sub(r'', '', md) md = re.sub(r'<script.?>.?</script>', '', md, flags=re.DOTALL) return md[:10000] truncate to safe length
What Undercode Say:
- Free tiers are a double‑edged sword: TinyFish lowers barriers for legitimate agent developers but also reduces cost for adversaries to build scraping bots and automated reconnaissance tools. Defenders must assume that attackers now have free, reliable web access.
- Markdown output is a game changer: By stripping HTML, JavaScript, and ads, TinyFetch reduces token counts by an order of magnitude – directly cutting LLM inference costs while improving context relevance.
- Integration with Code and OpenClaw hints at a future where every IDE agent has native, free web search. The mainstreaming of MCP servers will further commoditise web access, shifting competition toward retrieval quality rather than pricing.
Prediction:
Within 12 months, free web search and fetch APIs will become the default for all major agent frameworks, killing the “API key as a revenue stream” model for basic web access. This will accelerate the development of autonomous penetration testing agents, real‑time threat hunting bots, and self‑improving LLM workflows. However, it will also force website operators to implement stronger bot detection (e.g., Proof‑of‑Work challenges or AI‑specific CAPTCHAs) as the cost of scraping drops to zero. Security teams should prepare for a surge in AI‑driven reconnaissance – and start building their own free‑tier agents to monitor the digital perimeter continuously.
▶️ Related Video (76% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: The Web – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
