From Raw Logs to Real‑Time Intel: Build Your Own Linux Firewall Log Parser in Python + Video

By /

Listen to this Post

Featured Image

Introduction:

In the cybersecurity trenches, the difference between a contained nuisance and a full‑blown breach often comes down to visibility. Firewalls generate thousands of log lines, but raw telemetry is just noise until it is structured and analyzed. A recent home‑lab project by Rey Maldonado demonstrates this exact transition: building a Python‑based parser that ingests `iptables` logs from a Raspberry Pi to extract source/destination IPs, protocols, and ports. This article expands on that work, providing a complete guide to creating your own log parser, enriching the data, and setting the foundation for automated alerting—essential skills for any security analyst or network administrator.

Learning Objectives:

  • Understand how to configure `iptables` logging to capture network scan attempts.
  • Build a Python script to parse unstructured firewall logs into structured data.
  • Enrich parsed data with geolocation and threat intelligence feeds.
  • Implement JSON export and basic alerting logic for suspicious activity.

1. Configuring iptables for Detailed Logging

Before we can parse anything, we need to ensure our Linux firewall is generating the right logs. The standard `iptables` LOG target can be customized to include prefixes, which makes parsing easier and more reliable.

Step‑by‑step guide:

  1. Add logging rules: Insert rules at the top of the `INPUT` chain to log new connections (a common indicator of scans). 
    sudo iptables -I INPUT 1 -m state --state NEW -j LOG --log-prefix "SCAN_DETECTED: "
  2. Increase log verbosity (optional): To log more details like TCP flags, you can add specific rules. For example, to log all TCP SYN packets (the start of a TCP handshake, often used in port scans): 
    sudo iptables -I INPUT 2 -p tcp --syn -j LOG --log-prefix "SYN_PKT: "
  3. Verify the logs: The logs are typically written to `/var/log/kern.log` or `/var/log/messages` depending on your syslog configuration. Check them with: 
    sudo tail -f /var/log/kern.log | grep "SCAN_DETECTED"

You should see lines similar to:

kernel: [ 1234.5678] SCAN_DETECTED: IN=eth0 OUT= MAC=... SRC=192.168.1.100 DST=192.168.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=12345 PROTO=TCP SPT=54321 DPT=22 WINDOW=... RES=0x00 SYN URGP=0

2. Building the Core Parser in Python

The heart of the project is a Python script that reads these log lines and extracts key fields. We’ll use regular expressions to make the parsing robust.

Step‑by‑step guide:

  1. Set up the Python script: Create a file named fw_log_parser.py.
  2. Use a regex pattern: The iptables log format is relatively consistent. We can craft a regex to extract SRC, DST, PROTO, SPT, and DPT. 
    import re
import json</li>
</ol>

log_pattern = re.compile(
r'SRC=(?P<src_ip>\S+) DST=(?P<dst_ip>\S+) .? PROTO=(?P<proto>\S+)'
r'(?: SPT=(?P<spt>\d+))?(?: DPT=(?P<dpt>\d+))?'
)

parsed_logs = []

with open('/var/log/kern.log', 'r') as f:
for line in f:
if 'SCAN_DETECTED' in line or 'SYN_PKT' in line:
match = log_pattern.search(line)
if match:
log_entry = match.groupdict()
 Add a timestamp (simplified - you'd parse the actual syslog timestamp)
log_entry['raw_timestamp'] = line.split()[bash] + ' ' + line.split()[bash] + ' ' + line.split()[bash]
parsed_logs.append(log_entry)

print(json.dumps(parsed_logs, indent=2))

    This script reads the kernel log, filters for lines with our custom prefixes, applies the regex, and stores the results in a list of dictionaries.

    3. Structuring Output with JSON

    Moving from raw text to JSON is a game‑changer. JSON allows for easy integration with other tools, databases, and SIEM systems. We can enhance the previous script to write the parsed data to a file.

    Step‑by‑step guide:

    1. Modify the script for JSON export: Replace the `print` statement with a file write operation. 
      ... (previous code) ...</li>
</ol>

with open('parsed_firewall_logs.json', 'w') as json_file:
json.dump(parsed_logs, json_file, indent=2)

print(f"[+] Successfully parsed {len(parsed_logs)} log entries to parsed_firewall_logs.json")

      2. Run the script and inspect the output:

      python3 fw_log_parser.py
cat parsed_firewall_logs.json

      The output will be a clean JSON array, ready for ingestion by analysis tools.

      4. Enriching Data with Geolocation

      Raw IP addresses are useful, but knowing where a scan originates adds context. We can use a free API like `ip-api.com` to enrich our parsed logs with geolocation data (city, country, ISP).

      Step‑by‑step guide:

      1. Install the `requests` library: If not already installed. 
        pip install requests
      2. Add enrichment to the script: After parsing, loop through the entries and query the API. 
        import requests
import time</li>
</ol>

def get_geo_info(ip):
try:
response = requests.get(f'http://ip-api.com/json/{ip}', timeout=2)
if response.status_code == 200:
data = response.json()
if data['status'] == 'success':
return {
'country': data['country'],
'city': data['city'],
'isp': data['isp']
}
except:
pass
return None

... inside the main block, after building parsed_logs ...
enriched_logs = []
for entry in parsed_logs:
geo = get_geo_info(entry['src_ip'])
if geo:
entry.update(geo)
enriched_logs.append(entry)
time.sleep(0.1)  Be polite to the free API

Then export enriched_logs to JSON

        Note: For production use, consider a local GeoIP database like MaxMind’s GeoLite2 to avoid rate limits and latency.

        5. Implementing Alert Triggers

        The final step is turning analysis into action. We can add simple logic to flag “suspicious” activity—for example, a single source IP scanning multiple ports or hitting a sensitive port like SSH (22) or SMB (445).

        Step‑by‑step guide:

        1. Aggregate events by source IP: Count how many unique destination ports a source IP has targeted. 
          from collections import defaultdict

Group by source IP
source_activity = defaultdict(set)  Use a set to store unique destination ports</p></li>
</ol>

<p>for entry in enriched_logs:
src = entry['src_ip']
dpt = entry.get('dpt')
if dpt:
source_activity[bash].add(dpt)

Check for port scans (e.g., more than 5 unique ports)
alerts = []
for src, ports in source_activity.items():
if len(ports) > 5:
alerts.append({
'src_ip': src,
'unique_ports_scanned': list(ports),
'port_count': len(ports),
'severity': 'MEDIUM'
})

if alerts:
print("ALERT: Potential Port Scan Detected!")
print(json.dumps(alerts, indent=2))
 Here you could also send an email, write to a separate alert log, or trigger a webhook.

          6. Running as a Continuous Service

          Manually running the script isn’t practical. We can set it up to run periodically or tail the log in real time.

          Step‑by‑step guide (Real‑time tailing):

          1. Create a continuous version of the script: Use `subprocess` to `tail -f` the log file and process lines as they arrive. 
            import subprocess
import select</li>
</ol>

f = subprocess.Popen(['tail', '-F', '/var/log/kern.log'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p = select.poll()
p.register(f.stdout)

while True:
if p.poll(1):
line = f.stdout.readline().decode().strip()
 Process the line using the same parsing logic as before
if 'SCAN_DETECTED' in line:
match = log_pattern.search(line)
if match:
print("New event:", match.groupdict())
 Here you would trigger your alerting function immediately
else:
 No new line, maybe sleep briefly
time.sleep(0.5)

            This creates a lightweight, real‑time log monitor.

            7. Hardening and Linux/Windows Cross‑Reference

            While this guide focuses on Linux iptables, the principles apply universally. On a Windows system, you would use the `wevtutil` command or PowerShell to export Windows Firewall logs.

             Export Windows Firewall log (if logging is enabled)
Get-WinEvent -LogName 'Microsoft-Windows-Windows Firewall With Advanced Security/Firewall' -MaxEvents 100 | Export-Csv -Path firewall_logs.csv

            You could then write a similar Python parser for the CSV format. For cloud environments, you’d pull logs from AWS CloudWatch, Azure Monitor, or GCP Cloud Logging. The core idea—fetch, parse, structure, enrich, alert—remains identical, making this a transferable skill across platforms.

            What Undercode Say:

            • From Noise to Signal: The single most important skill in defensive security is the ability to transform verbose, unstructured machine data into a structured format that can be queried and understood. This project is a microcosm of that larger discipline.
            • Detection is a Pipeline: Alerting isn’t magic; it’s the end result of a data pipeline: generation → collection → parsing → enrichment → correlation. Building this pipeline yourself, even in a lab, demystifies how commercial SIEM and NDR solutions operate.

            Prediction:

            As network perimeters continue to dissolve with remote work and cloud adoption, the value of host‑based and firewall log analysis will not diminish but rather shift in context. We predict a rise in “edge‑native” security tools that combine the simplicity of this Python parser with the scale of cloud functions (e.g., AWS Lambda). The future belongs to security professionals who can glue together open‑source tools and APIs to create custom detection logic, moving away from black‑box appliances toward transparent, code‑defined security operations.

            ▶️ Related Video (78% Match):

            🎯Let’s Practice For Free:

            IT/Security Reporter URL:

            Reported By: Rmaldo Github – Hackers Feeds
            Extra Hub: Undercode MoN
            Basic Verification: Pass ✅

            🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

            💬 Whatsapp | 💬 Telegram

            📢 Follow UndercodeTesting & Stay Tuned:

            𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: