Listen to this Post
Introduction:
In the world of distributed systems and heterogeneous server fleets, centralized monitoring is the backbone of operational security and efficiency. While dashboards often focus on real-time metrics, the true value lies in ad-hoc, cross-host data collection that can identify anomalies, such as unauthorized file dumps or disk space hogs. By leveraging `zabbix_get` in a loop, we transform a simple diagnostic tool into a powerful engine for fleet-wide intelligence gathering, bridging the gap between monitoring and active system administration.
Learning Objectives:
- Understand how to use `zabbix_get` for active data retrieval from remote Zabbix agents.
- Master the syntax for aggregating command outputs across multiple hosts.
- Learn to automate the collection of forensic data (like large file lists) for security auditing.
You Should Know:
1. Preparing the Remote Hosts: The Cron Job
Before we can retrieve data, we need to ensure the data exists on the remote endpoints. The original post highlights a cron job that generates a list of the top five largest files on a system. From a cybersecurity perspective, this is crucial for identifying potential data exfiltration staging areas or rogue log files.
Step‑by‑step guide:
1. SSH into the target Linux host.
- Create the script. Open a crontab editor: `crontab -e`
3. Add the following line to run every hour:0 find / -type f -exec du -h {} + 2>/dev/null | sort -hr | head -n 5 > /tmp/top.txt
4. What this does:
find / -type f: Locates all files starting from the root directory.-exec du -h {} +: Executes `du` (disk usage) in human-readable format on the found files.2>/dev/null: Suppresses permission denied errors (essential for security, as you don’t want errors revealing system structure to unauthorized local users).sort -hr: Sorts the results by size in human-readable format, descending.head -n 5: Takes only the top 5.> /tmp/top.txt: Writes the output to a temporary file.
2. The Zabbix Server Script: Looping for Data
With the data generated on each client, the Zabbix server can now pull it. The magic happens with a simple bash loop that iterates through your host list.
Step‑by‑step guide:
- Create a host list file on your Zabbix server, e.g.,
hosts.txt, containing one hostname per line.web-server-01 db-server-01 cache-server-01
2. Write the retrieval script:
!/bin/bash OUTPUT_FILE="/tmp/aggregated_fleet_files.html" echo "<html><body> <h1>Top 5 Largest Files per Host</h1> " > $OUTPUT_FILE while read host; do echo " <h2>Host: $host</h2> <pre>" >> $OUTPUT_FILE Use zabbix_get to retrieve the content of the remote file zabbix_get -s $host -k "vfs.file.contents[/tmp/top.txt]" >> $OUTPUT_FILE 2>/dev/null echo "</pre> " >> $OUTPUT_FILE done < hosts.txt echo "</body></html>" >> $OUTPUT_FILE echo "Report generated at $OUTPUT_FILE"
3. Explanation:
zabbix_get -s $host: Specifies the target host.-k "vfs.file.contents[/tmp/top.txt]": This is the key. It uses Zabbix’s built-in `vfs.file.contents` item to remotely read the file we created via cron.- The loop wraps each host’s output in HTML `
` tags for formatting.
3. Adding HTML Formatting for Readability
Raw data is useful, but formatted data is presentable. As hinted in the original post, applying HTML formatting transforms a log dump into a report suitable for security briefings or management reviews. The script above already implements this by wrapping the output in basic HTML, but we can enhance it.Enhanced Formatting Logic:
Modify the loop to handle cases where data retrieval fails (e.g., host down, file missing).Inside the while loop DATA=$(zabbix_get -s $host -k "vfs.file.contents[/tmp/top.txt]" 2>&1) if [[ $DATA == "ZBX_NOTSUPPORTED" ]] || [[ $DATA == "Timeout" ]]; then echo " <h2>Host: $host</h2> <p style='color:red;'>Error: Unable to retrieve data. Agent or file may be unavailable.</p> " >> $OUTPUT_FILE else echo " <h2>Host: $host</h2> <pre>$DATA</pre> " >> $OUTPUT_FILE fi
- Automating with a Full Script and Command List
For a real-world Security Operations Center (SOC) workflow, this script should be automated. Here is a complete, production-ready version with error handling and timestamping.
Full Automation Script:
!/bin/bash Fleet File Collector for Security Auditing Author: IT Engineering Team Date: $(date +%Y-%m-%d) HOST_LIST="/etc/zabbix/scripts/fleet_hosts.txt" REPORT_DIR="/var/www/html/reports" REPORT_FILE="$REPORT_DIR/fleet_report_$(date +%Y%m%d_%H%M).html" Ensure report directory exists mkdir -p $REPORT_DIR Start HTML Report cat <<EOF > $REPORT_FILE <html> <head><title>Fleet File Audit</title></head> <body> <h1>Top 5 Files per Host - $(date)</h1> EOF Loop through hosts while read host; do Skip empty lines [ -z "$host" ] && continue echo " <h2>Host: $host</h2> <pre>" >> $REPORT_FILE Attempt to fetch data with a timeout if zabbix_get -s $host -k "vfs.file.contents[/tmp/top.txt]" -t 5 >> $REPORT_FILE 2>/dev/null; then echo "</pre> " >> $REPORT_FILE else echo "Failed to connect or retrieve file." >> $REPORT_FILE echo "</pre> <p style='color:red;'>[bash] Connection failed or agent unavailable.</p> " >> $REPORT_FILE fi done < $HOST_LIST Close HTML echo "</body></html>" >> $REPORT_FILE Display result echo "[bash] Report generated: $REPORT_FILE"
5. Security Implications: Why This Matters
In cybersecurity, "Know Your Environment" is paramount. This technique allows an engineer to proactively look for anomalies. If a compromised host begins to aggregate data for exfiltration, the `find` command might unexpectedly return a massive file in a user's temp directory (e.g., database_dump.zip).
Step‑by‑step guide to using this for threat hunting:
- Baseline: Run the script once a day for a week. Learn what the "Top 5" files usually are on each server (e.g., log files, database files).
- Alert on Change: Modify the script to compare the output against a baseline using
diff. - Investigate: If a new, unusually large file appears (like a `.tar` or `.7z` archive in
/tmp), it could indicate a hacker packing data for exfiltration.
6. Extending to Windows Environments
Zabbix is cross-platform. The same concept applies to Windows servers using PowerShell and Zabbix agents.
On Windows Client (PowerShell Script via Task Scheduler):
Get-ChildItem C:\ -Recurse -ErrorAction SilentlyContinue |
Where-Object { -not $<em>.PSIsContainer } |
Sort-Object Length -Descending |
Select-Object -First 5 |
Format-Table Name, @{Name="Size(GB)";Expression={[bash]::Round($</em>.Length/1GB,2)}} -AutoSize |
Out-File -FilePath C:\temp\top.txt
On the Zabbix server, you would use the same `zabbix_get -s
What Undercode Say:
- Centralized Visibility: Using `zabbix_get` in a loop transforms passive monitoring into active, actionable reconnaissance, allowing engineers to "see" into every server without manual SSH logins.
- Security Through Automation: Automating the collection of disk usage data is a low-effort, high-reward method for detecting early indicators of compromise, such as unauthorized data staging.
- Scalable Forensics: This method provides a scalable way to perform basic forensic triage across hundreds of servers, identifying outliers that warrant deeper investigation.
Prediction:
As infrastructure grows more ephemeral (containers, serverless), the need for agentless or API-driven data aggregation will increase. However, for traditional server fleets, techniques like this will evolve to integrate directly with SIEM (Security Information and Event Management) systems, feeding raw host data into machine learning models to automatically detect volumetric anomalies, shifting the burden from manual script execution to automated threat detection.
▶️ Related Video (72% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Giedrius Stasiulionis - Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
