Listen to this Post
Introduction:
Security Information and Event Management (SIEM) platforms are critical for modern security operations, but their immense cost is often driven by the storage of low-value, noisy log data. By strategically filtering out routine, non-threatening events before ingestion, organizations can achieve massive cost savings and significantly improve query performance, redirecting resources to higher-value security initiatives.
Learning Objectives:
- Understand the core principles of pre-ingestion log filtering and optimization.
- Learn specific commands and techniques to identify and eliminate noisy data from common sources like Okta.
- Develop a strategy for compliant, cost-effective log retention outside the SIEM.
You Should Know:
1. Identifying and Filtering Okta Sign-Out Events
A primary source of SIEM noise is authentication logs. Events like successful sign-outs, while necessary for compliance, offer minimal detection value. Filtering them at the source prevents them from consuming expensive SIEM storage.
Okta System Log API Call (Example):
`GET https://{yourOktaDomain}/api/v1/logs?filter=eventType eq “user.session.end”`
Step-by-Step Guide:
This API call fetches a list of all user session end (sign-out) events from the Okta system log. The goal is to identify the volume and structure of these events. In a pre-processing script or within your log forwarder (e.g., a Splunk Universal Forwarder, Fluentd, or Logstash configuration), you would implement a filter to drop events where `eventType` equals `”user.session.end”` before they are sent to the SIEM. This prevents them from ever entering your costly ingestion pipeline.
2. Removing Verbose Fields from Windows Security Logs
The Windows Security Log contains crucial security data but is often bloated with redundant or unnecessary fields that increase storage costs without adding analytical value.
Windows PowerShell Command (Example):
`Get-WinEvent -LogName Security | Where-Object {$_.Id -ne 4634} | Select-Object -Property TimeCreated, Id, ProviderName, UserId, MachineName`
Step-by-Step Guide:
This PowerShell command retrieves events from the Security log, filters out event ID 4634 (a detailed logoff event, often verbose and low-value for immediate detection), and then selects only a specific set of crucial properties for output. To implement this as a filter, you would configure your Windows log forwarder (e.g., Winlogbeat or the Splunk Universal Forwarder) to apply a transform that excludes unwanted event IDs and strips out pre-defined verbose fields (like lengthy XML blobs or task categories) before transmission.
3. Linux Auditd Rule to Omit Common Noise
The Linux Audit daemon is powerful but can generate excessive noise. Pruning common, expected events drastically reduces volume.
Linux Auditd Rule (Example):
`-a exclude,always -F msgtype!=DAEMON_START -F msgtype!=DAEMON_END -F msgtype!=SYSCALL -F msgtype!=USER_ACCT`
Step-by-Step Guide:
This rule is placed in the audit.rules configuration file (/etc/audit/audit.rules). It tells the audit daemon to exclude (not log) events that are of the type DAEMON_START, DAEMON_END, and `USER_ACCT` from being written to the audit log. These are typically routine, non-security-critical events. By preventing them from being written to disk, your log collection agent (e.g., Osquery, Splunk UF) never picks them up, ensuring they never reach the SIEM. Always test rules in a monitoring mode first.
4. Splunk Forwarder Transformation for Pre-Ingestion Filtering
Splunk Universal Forwarders can be configured with `props.conf` and `transforms.conf` files to drop events or fields at the source.
Splunk transforms.conf Configuration (Example):
`[bash] REGEX = eventType”:”user\.session\.end DEST_KEY = queue FORMAT = nullQueue`
Step-by-Step Guide:
This configuration stanza creates a transformation named okta_drop_signouts. The `REGEX` statement looks for the JSON string indicating a sign-out event. When a log entry matches this pattern, the transformation instructs the forwarder to send it to the `nullQueue` – effectively deleting it before it is sent to the indexer. This is a powerful method for implementing pre-ingestion filtering directly within a common log forwarding infrastructure.
5. Syslog-ng Filter to Block Health Check Pings
Application and network devices constantly generate health check pings and routine status updates that clutter the SIEM.
Syslog-ng Configuration Snippet (Example):
`filter f_drop_healthchecks { not match(“GET /healthcheck HTTP/1.1” value(“MESSAGE”)) and not match(“ICMP Echo Reply” value(“MESSAGE”)); }; log { source(s_network); filter(f_drop_healthchecks); destination(d_siem); };`
Step-by-Step Guide:
In your `syslog-ng.conf` file, you define a filter (f_drop_healthchecks) that uses regex to match on common health check patterns in the log message. The `not` operator ensures any message containing these patterns is excluded. The `log` path then applies this filter to the incoming log source (s_network). Only events that do not match the health check patterns are forwarded to the SIEM destination (d_siem).
6. Storing Filtered Logs in Cost-Effective Cold Storage
Compliance often requires retaining filtered logs, but they don’t need to be in the hot SIEM storage. Object storage like AWS S3 Glacier is ideal.
AWS CLI Command to Archive a Log File (Example):
`aws s3 cp /path/to/filtered-logfile.log s3://your-compliance-bucket/archive/ –storage-class GLACIER`
Step-by-Step Guide:
After your filtering mechanism writes low-value logs (e.g., Okta sign-outs) to a local file, you can use a scheduled script or tool with this AWS CLI command to transfer them to an S3 bucket. The `–storage-class GLACIER` parameter ensures the data is stored at the lowest possible cost while remaining available for forensic or compliance audits if needed. This satisfies retention requirements without burdening the SIEM.
7. Calculating Potential Cost Savings
Understanding the financial impact is key to justifying this optimization work. The calculation is straightforward.
Bash Command to Calculate Daily Log Volume (Example):
`grep “eventType.user.session.end” okta_logs.json | wc -l | awk ‘{print $1 0.5 365 / 1024}’`
Step-by-Step Guide:
This command pipeline estimates annual savings. It: 1) counts the number of sign-out events in a sample log file (wc -l), 2) multiplies that by an estimated average log size in KB (e.g., 0.5KB), 3) multiplies by 365 days, and 4) converts to MB (/1024). You can then multiply the resulting annual volume by your SIEM’s cost per GB/month. This provides a data-driven basis for your cost-saving proposal.
What Undercode Say:
- Shift Left on Log Filtering: The greatest efficiency gains are achieved by filtering noise as close to the source as possible, not within the SIEM query. This is a fundamental mindset shift from “collect everything” to “curate intentionally.”
- Compliance is Not Compromised: Cost optimization does not mean deleting data required for audits. It means architecting a tiered storage strategy, keeping high-fidelity detection data hot and moving low-value compliance data to cold storage, which is both compliant and financially prudent.
Analysis: The post highlights a critical maturity evolution in security engineering. The initial “collect it all” SIEM approach is no longer sustainable due to escalating cloud data costs and increasingly verbose logging. The guidance provided moves the practice from a reactive cost center to a proactive, financially-aware engineering discipline. By treating log data like any other data pipeline—where quality and value are prioritized over sheer volume—teams can dramatically improve performance and reallocate significant budget towards talent and tools that directly improve security posture. This is not a corner-cutting measure but a strategic optimization essential for modern, scalable security operations.
Prediction:
The rising cost of data ingestion and storage will force a paradigm shift in SIEM and security data lake architectures. We will see the rapid adoption of intelligent, pre-ingestion filtering agents and the bundling of these capabilities directly into log source platforms (like Okta) and forwarder software. “Log curation” will become a standard security engineering skill, and the market will value SIEM solutions that provide transparent, granular cost controls and seamless integration with low-cost cold storage providers, moving the industry towards a more efficient and sustainable data model.
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Darwin Salazar – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
