Listen to this Post
Introduction:
As organizations migrate to cloud-1ative SIEM solutions like Microsoft Sentinel, the exponential growth of log data presents a dual challenge: maintaining robust security visibility while controlling escalating ingestion costs. Traditional approaches of ingesting all firewall, Entra ID, and network appliance logs into expensive Analytics tiers are no longer sustainable. This article explores native Azure capabilities—Data Collection Rules (DCRs), Workspace Transformations, and custom tables—that enable security teams to filter, split, and route logs into cost-effective Basic tiers, achieving savings of 35-40% without sacrificing investigative integrity.
Learning Objectives:
- Understand how to leverage Azure Monitor’s native ingestion pipeline to filter and split logs into multiple Sentinel tables.
- Master the deployment of Azure Monitor Agent (AMA) with Syslog/CEF collectors for firewall log ingestion.
- Implement custom table creation and DCR transformations to route data to Basic tiers.
- Apply cost-optimization strategies for Entra ID diagnostic settings and firewall logs.
- Automate table schema creation using PowerShell tools for streamlined deployments.
- The Cost Dilemma: Why Your Current SIEM Ingestion Strategy Is Breaking the Bank
In traditional on-premises SIEM deployments, log ingestion costs were largely fixed capital expenses. However, in cloud-1ative SIEMs like Microsoft Sentinel, every gigabyte ingested translates directly to operational expenditure. The default configuration for Entra ID (formerly Azure AD) diagnostic settings routes all logs to the Analytics tier—the most expensive option. Similarly, firewall and network appliance logs, often ingested “just in case” or for compliance reasons, consume massive volumes of high-cost storage.
The problem is compounded by the fact that not all log data holds equal security value. While threat detection and hunting require real-time, high-performance Analytics-tier data, raw traffic logs or verbose audit trails may only be needed for periodic investigations or compliance audits. The key insight is that you don’t need to eliminate this data—you just need to store it intelligently.
Step‑by‑step guide to assessing your current ingestion costs:
- Audit your Sentinel workspace: Navigate to Log Analytics Workspace > Usage and estimated costs to identify which tables are driving the highest ingestion volumes.
- Identify high-volume, low-value sources: Focus on `SigninLogs` (Entra ID), `CommonSecurityLog` (firewall/CEF), and any custom tables ingesting verbose network data.
- Determine retention requirements: For compliance data, Basic tier with longer retention may be more cost-effective than Analytics tier with shorter retention.
- Calculate potential savings: Use Azure Pricing Calculator to compare Analytics vs. Basic tier costs for your estimated monthly ingestion volume.
-
Deploying the Linux Collector: The Foundation of Firewall Log Ingestion
Before you can filter and split logs, you need a reliable collection mechanism. For firewall and CEF logs, the standard approach involves deploying a Linux-based collector running the Azure Monitor Agent (AMA) with the Syslog/CEF collector enabled.
The AMA represents a significant upgrade over the legacy Log Analytics Agent, offering improved performance, native filtering capabilities, and seamless integration with Data Collection Rules (DCRs). Unlike third-party solutions like Logstash, which require complex custom parsing rules, AMA leverages Azure’s native transformation capabilities to normalize and route data without additional overhead.
Step‑by‑step guide to deploying the Linux collector:
- Provision a Linux VM: Deploy an Ubuntu 20.04 LTS or RHEL 8.x VM in the same region as your Sentinel workspace. Ensure it has adequate CPU (minimum 2 vCPUs) and memory (4 GB RAM) for high-volume log processing.
2. Install the Azure Monitor Agent:
Download and install the AMA package wget https://aka.ms/azuremonitoragent-linux -O ama.deb sudo dpkg -i ama.deb
3. Configure Syslog/CEF collection: Edit `/etc/rsyslog.d/50-default.conf` to forward firewall logs to the local AMA endpoint:
Forward all CEF-formatted logs to AMA . @127.0.0.1:25224
4. Restart rsyslog and verify:
sudo systemctl restart rsyslog sudo systemctl status azuremonitoragent
5. Associate the VM with your Sentinel workspace using the Azure Portal or ARM templates.
- Creating Custom Tables: The Key to Basic Tier Ingestion
One of the less obvious aspects of Microsoft Sentinel is that data cannot be directly ingested into the Basic tier using default diagnostic settings. Instead, you must create custom tables specifically designed to receive filtered logs at the Basic tier. This is where the `tableCreator` PowerShell tool, developed by Marko Lauren, becomes invaluable.
The tool captures the schema from existing Sentinel tables (e.g., `SigninLogs` or CommonSecurityLog) and creates new custom tables with identical schemas, but configured for the Basic tier. This approach eliminates manual schema definition errors and ensures seamless compatibility with existing KQL queries.
Step‑by‑step guide to creating custom tables using PowerShell:
1. Install the `tableCreator` tool from GitHub:
git clone https://github.com/markolauren/sentinel.git cd sentinel/tableCreator\ tool
2. Authenticate to Azure:
Connect-AzAccount -Subscription "Your-Subscription-ID"
3. Run the tool to clone an existing table schema:
./CreateCustomTable.ps1 -WorkspaceName "YourLogAnalyticsWorkspace" ` -SourceTable "CommonSecurityLog" ` -1ewTableName "FirewallLogs_Basic" <code>-Tier "Basic"
4. Verify the new custom table appears in your Log Analytics workspace under Tables.
5. Optional: Use the tool to clone multiple tables simultaneously for different log sources (e.g.,SigninLogs_Basic,NetworkLogs_Basic`).
- Data Collection Rules (DCRs): The Brains of the Operation
Data Collection Rules (DCRs) are the central orchestration mechanism that defines how data flows from your sources into Sentinel tables. By modifying the DCR associated with your AMA collector, you can apply filtering logic and route specific subsets of data to different tables—all without writing a single line of custom parsing code.
The power of DCRs lies in their ability to use KQL-like transformations to evaluate incoming log entries and direct them based on predefined conditions. For example, you can route all firewall logs containing “DENY” or “BLOCK” actions to the Analytics tier for real-time threat detection, while routing “ALLOW” or “INFO” logs to the Basic tier for compliance storage.
Step‑by‑step guide to creating and modifying a DCR:
- Create a new DCR in the Azure Portal: Navigate to Monitor > Data Collection Rules > Create.
- Define data sources: Select Linux Syslog or CEF as the source type, and specify the log levels to collect (e.g., `LOG_INFO` through
LOG_EMERG).
3. Configure transformation:
{
"transform": "source | where SeverityLevel != 'INFO' | project-away Field1, Field2"
}
This example filters out informational logs and removes unnecessary fields before ingestion.
4. Set destinations: Add two destinations—one for the Analytics-tier table (e.g., CommonSecurityLog) and one for the Basic-tier custom table (e.g., FirewallLogs_Basic).
5. Define data flow: Specify that logs matching the transformation rule should be sent to both destinations, or split them based on conditions.
6. Associate the DCR with your Linux VM and monitor the ingestion pipeline for errors.
- Filtering and Splitting Entra ID Logs: A Cost-Saving Blueprint
Entra ID logs present a unique challenge due to their sheer volume. Default diagnostic settings route all sign-in logs, audit logs, and provisioning logs to the Analytics tier. However, many of these logs—particularly successful sign-ins from trusted IP ranges—hold limited threat detection value and can be safely moved to the Basic tier.
The solution involves a two-pronged approach: first, use workspace transformations to filter out low-value events at ingestion time; second, route the remaining high-value events to custom Basic-tier tables while keeping critical security events (e.g., failed sign-ins, risky sign-ins, privilege escalations) in the Analytics tier.
Step‑by‑step guide to filtering Entra ID logs:
- Review your Entra ID diagnostic settings: Ensure logs are being sent to your Log Analytics workspace.
- Create a custom table for Basic-tier Entra ID logs using the `tableCreator` tool:
./CreateCustomTable.ps1 -WorkspaceName "YourWorkspace" ` -SourceTable "SigninLogs" ` -1ewTableName "SigninLogs_Basic" ` -Tier "Basic"
- Create a workspace transformation to filter out low-value events:
source | where ResultType != "0" // Keep only failed sign-ins | where IPAddress !in (ipv4_is_in_any_range("10.0.0.0/8", "192.168.0.0/16")) - Update your Data Collection Rule to apply this transformation before routing to the custom table.
- Monitor the ingestion volume over 48 hours to validate cost savings.
6. Automating Deployment with PowerShell and ARM Templates
Manual configuration of DCRs, custom tables, and transformations is time-consuming and error-prone, especially in large-scale enterprise environments. To streamline deployment, Marko Lauren’s `tableCreator` PowerShell tool and ARM template-based automation frameworks provide reusable, version-controlled infrastructure-as-code solutions.
By automating these deployments, security teams can rapidly provision cost-optimized Sentinel environments across multiple subscriptions and regions, ensuring consistent configurations and reducing operational overhead.
Step‑by‑step guide to automating Sentinel log optimization:
- Package your DCR and transformation rules as ARM templates:
{ "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json", "resources": [ { "type": "Microsoft.OperationalInsights/workspaces/tables", "name": "[concat(parameters('workspaceName'), '/', parameters('tableName'))]", "properties": { "schema": { "name": "[parameters('tableName')]", "columns": [] } } } ] } - Use PowerShell to deploy the template across multiple workspaces:
$params = @{ workspaceName = "Sentinel-Prod" tableName = "FirewallLogs_Basic" } New-AzResourceGroupDeployment -ResourceGroupName "Sentinel-RG" ` -TemplateFile "./customTableTemplate.json" ` -TemplateParameterObject $params - Implement CI/CD pipelines using Azure DevOps or GitHub Actions to automate template validation and deployment.
- Schedule regular audits to identify new log sources that can be optimized using the same patterns.
What Undercode Say:
- Key Takeaway 1: Native Azure capabilities—DCRs, workspace transformations, and custom tables—provide a vendor-1ative, no-code approach to optimizing Sentinel ingestion costs, eliminating the need for third-party log shippers.
- Key Takeaway 2: A 35-40% reduction in firewall log ingestion costs is achievable by splitting logs between Analytics and Basic tiers based on security value, without compromising threat detection capabilities.
Analysis:
Marko Lauren’s contributions to the Microsoft Sentinel community, particularly through the `tableCreator` PowerShell tool and detailed technical posts, highlight a critical gap in cloud SIEM adoption: cost management. Many organizations blindly ingest all logs into expensive tiers, only to be shocked by their monthly bills. Lauren’s approach is both elegant and practical—it leverages Azure’s native capabilities to achieve cost savings without sacrificing security visibility. The key insight is that not all logs are created equal, and intelligent routing based on log content and security relevance is the future of sustainable SIEM operations. For security architects, this represents a shift from “ingest everything” to “ingest intelligently,” a mindset that will become increasingly critical as data volumes continue to explode.
Prediction:
- +1 The adoption of intelligent log routing and tiered storage will become a standard best practice for all cloud SIEM deployments within the next 18 months, driven by both cost pressures and regulatory demands for long-term data retention.
- +1 Microsoft will continue to enhance native filtering and transformation capabilities, potentially introducing AI-driven recommendations for optimal log routing based on threat intelligence and usage patterns.
- -1 Organizations that fail to implement log optimization strategies will face unsustainable cost increases, potentially forcing them to reduce log retention periods or disable critical log sources—creating dangerous security blind spots.
- +1 The open-source community will develop additional automation tools and KQL query libraries to further simplify the deployment of cost-optimized Sentinel environments, accelerating adoption across all enterprise sizes.
▶️ Related Video (78% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Markolauren Sentinel – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
