What Logs Should Be Ingested For Effective Security Monitoring?

When setting up a SIEM (Security Information and Event Management) system, one of the most common questions is: “What logs should we ingest?” The humorous but practical answer is: “All logs except WAF (Web Application Firewall) logs!” While this may sound extreme, the reality is that comprehensive log collection is crucial for threat detection, incident response, and forensic investigations.

You Should Know: Essential Logs for Security Monitoring

1. Endpoint Logs (Windows & Linux)

Endpoint logs provide visibility into user and system activities. Key logs include:

Windows Event Logs

Security Logs (Event ID 4624, 4625, 4688): Track logins, failed attempts, and process execution.
```
Get-WinEvent -LogName Security -MaxEvents 10 | Format-Table -AutoSize
```

System Logs: Detect service crashes, driver failures, and unexpected reboots.

Get-WinEvent -LogName System | Where-Object {$_.Level -eq 2} | Select-Object -First 5

PowerShell Logs: Critical for detecting malicious scripts.

Enable-PSRemoting -Force 
Set-ExecutionPolicy RemoteSigned

Linux Syslog & Auditd

/var/log/auth.log: Tracks authentication events (SSH, sudo).
```
grep "Failed password" /var/log/auth.log
```

Auditd Logs: Monitor file access, user commands, and system calls.

sudo auditctl -l  List active rules 
sudo ausearch -k mykey -i  Search audit logs

2. Network Logs (Firewall, IDS/IPS, NetFlow)

Firewall Logs: Detect unusual traffic patterns.

journalctl -u firewalld --no-pager | grep "DROP"

Suricata/Snort Alerts: Analyze intrusion attempts.
```
tail -f /var/log/suricata/fast.log
```
NetFlow/sFlow: Identify lateral movement.

3. Cloud & Application Logs

AWS CloudTrail: Track API calls in AWS.

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket

Docker/Kubernetes Logs: Monitor containerized environments.
```
kubectl logs -f <pod-name> --namespace=<namespace>
```

Authentication & Identity Logs (Active Directory, Okta, Duo)

– AD Logs: Detect brute-force attacks.

Get-EventLog -LogName Security -InstanceId 4625 -After (Get-Date).AddHours(-24)

– Okta/Duo Logs: Review MFA bypass attempts.

5. Why Exclude WAF Logs?

WAF logs are often noisy and less useful for threat hunting. Instead, focus on:
– Blocked attack patterns (SQLi, XSS).
– False positives that may indicate evasion attempts.

What Undercode Say

Effective log ingestion is the backbone of Detection Engineering and Threat Hunting. While collecting all logs is ideal, prioritize:
– Security-relevant logs (auth, process execution, network traffic).
– High-value targets (AD, cloud APIs, critical servers).
– Logs with low noise but high signal (auditd, PowerShell transcripts).

Automate log parsing with Elasticsearch, Splunk, or Sigma rules to reduce manual effort.

Prediction

As attackers evolve, log sources will expand (IoT, AI-driven systems). Future SIEMs will rely on ML-based log filtering to prioritize critical alerts.

Expected Output:

 Sample SIEM log ingestion pipeline 
1. Collect: Rsyslog/Filebeat → Logstash 
2. Parse: Grok filters for structured data 
3. Store: Elasticsearch cluster 
4. Alert: Sigma rules → SIEM alerts

Relevant URLs:

References:

Reported By: Inode Waf – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

Join Our Cyber World:

💬 Whatsapp | 💬 Telegram

Listen to this Post