Listen to this Post
Introduction
Microsoft Sentinel users have long struggled with a frustrating blind spot: understanding exactly how their log ingestion costs are distributed across different data plans. This lack of visibility often led to unexpected bills and required custom coding to bridge the gap. The security community just got two major (but unannounced) updates from Microsoft that aim to eliminate these challenges by enhancing cost transparency directly within the platform, particularly for the new Data Lake tier.
Learning Objectives
- Understand the impact of the new `Plan` field in the Log Analytics Usage table on cost analysis.
- Learn how the relaxed permissions for the Cost Management blade empower SOC teams without administrative overhead.
- Master practical KQL queries and Azure CLI commands to monitor, analyze, and optimize your Microsoft Sentinel and Defender data ingestion costs.
You Should Know
- The `Plan` Field Is Finally Here, Putting an End to Guesswork
For years, security teams had to perform complex custom logic to understand the cost distribution of their data ingestion across different tiers like Analytics, Basic, or the new Data Lake. This long-standing gap has been closed. The Usage table within your Log Analytics Workspace now includes a direct and native `Plan` field.
This small addition is a game-changer for financial governance. It allows you to immediately see which data is being ingested into which plan directly from your central billing table. This makes cost analysis and reporting dramatically more straightforward and accurate, removing the need for manual enrichment or workarounds.
Step‑by‑step guide explaining what this does and how to use it:
The `Usage` table is a built-in system table in every Log Analytics workspace that acts as your log ingestion ledger. It tracks how much data is ingested, by which solution and, crucially now, into which table plan. To use this for cost analysis, you will use KQL (Kusto Query Language).
Step 1: Connect to your Log Analytics Workspace. Navigate to your Sentinel workspace in the Azure portal and open the “Logs” blade.
Step 2: Run a Basic Cost Distribution Query. Paste the following KQL query to see how your billable data is spread across different plans for the last 90 days:
Usage | where TimeGenerated > ago(90d) | where IsBillable == true | summarize TotalIngestGB = sum(Quantity) / 1000 by Plan, Solution | order by TotalIngestGB desc
Step 3: Identify Top Data Sources per Plan. To see which specific tables are generating the most costs for each plan, use this query:
Usage | where TimeGenerated > ago(30d) | where IsBillable == true | summarize GB_Ingested = sum(Quantity) / 1000 by Plan, DataType | order by GB_Ingested desc
Step 4: Use Azure CLI (Alternative). For programmatic access to the `Usage` table data, you can use Azure CLI to query Log Analytics:
az monitor log-analytics query --workspace "YOUR_WORKSPACE_NAME" --analytics-query "Usage | where TimeGenerated > ago(7d) | where IsBillable == true | summarize GB_ingested=sum(Quantity)/1000 by Plan" --out table
This will output a table showing the sum of billable ingestion in GB grouped by the `Plan` field, providing a quick high-level view of your cost drivers for the past week.
- Cost Management Is Now Accessible Without Billing Admin Role
A significant operational hurdle for security teams has been removed. Before this update, accessing the Cost Management blade to view cost and ingestion data required a user to have a combination of Security Administrator and Billing Administrator permissions, leading to friction and delays. This is no longer the case.
This improvement means that your security operations teams can now gain vital financial insights without being granted elevated billing permissions. For organizations leveraging the Microsoft Sentinel and Microsoft Defender Data Lake, this is critical. It allows SOC analysts and engineers to understand data growth, retention, and cost trends in real-time, empowering them to optimize data ingestion without bureaucratic overhead.
Step‑by‑step guide explaining what this does and how to use it:
The Microsoft Sentinel Cost Management experience is now a native feature in the Microsoft Defender portal. It is designed specifically to give you granular cost control over your Data Lake tier, complementing the broader Azure Cost Analysis tools.
Step 1: Access the Sentinel Cost Management Blade. Log into the Microsoft Defender portal. Navigate to Microsoft Sentinel > Cost management. This view is specifically for the Data Lake tier and provides detailed insights that you previously might not have had access to.
Step 2: Analyze Data Lake Ingestion. Within the “Usage” tab, you will find interactive reports. The “Data lake ingestion” report shows the total data ingested in GB, a trendline over time, and a list of the top 10 tables by ingestion volume. This helps you immediately pinpoint the most data-intensive tables in your data lake.
Step 3: Track Query and Compute Costs. The “Data lake query” view displays the total amount of data scanned in GB, for which you are charged. Similarly, “Advanced data insights” shows compute hours used for notebooks or jobs. By analyzing these, you can identify inefficient queries or resource-intensive jobs that are driving up costs.
Step 4: Configure Proactive Cost Alerts. Go to the “Notifications” page within the Cost Management blade. You can set usage thresholds (e.g., an alert when Data Lake Query scanned data exceeds 1000GB) and receive email alerts when those limits are reached. This is a critical step to avoid unexpected bills.
Step 5: Correlate with Azure Cost Analysis. For a holistic view, you can also use Azure’s native Cost Management. Since the Sentinel Data Lake is exposed as an Azure resource, all its meters are available in Azure portal > Cost Management + Billing > Cost Analysis. You can filter by the “Sentinel” service name to cross-reference the data found in the Defender portal.
- Taming the Five New Data Lake Meters with KQL and PowerShell
With the introduction of the Data Lake, your Sentinel bill now includes five new meters: Data Lake Ingestion, Storage, Query, Advanced Data Insights, and Data Processing. To avoid a shocking bill, you need to monitor these proactively.
Step‑by‑step guide explaining what this does and how to use it:
Step 1: Identify Your Most Expensive Tables. Run the following KQL query in your Sentinel Log Analytics workspace to find the noisiest log sources:
union withsource=SourceTable | where TimeGenerated > ago(7d) | summarize TotalSizeGB = sum(_BilledSize) / (1024 1024 1024) by SourceTable | order by TotalSizeGB desc
This query scans all tables over the last 7 days and sums the `_BilledSize` (in bytes) for each table, presenting the top contributors to your analytics-tier bill.
Step 2: Change a Table Plan Programmatically. You can change a table’s plan from “Analytics” to “Basic” or “Auxiliary” (Data Lake) using PowerShell to save costs on high-volume logs that don’t need real-time detection. You can even use open-source scripts like `tableSettings.ps1` to apply these changes at scale across many tables. To set a table plan using PowerShell:
Example to set a custom table to Auxiliary (Data Lake) plan
$resourceId = "/subscriptions/your-subscription-id/resourceGroups/your-resource-group/providers/Microsoft.OperationalInsights/workspaces/your-workspace-name/tables/Your_Table_CL"
$properties = @{
plan = "Auxiliary"
}
Invoke-AzRestMethod -Path ("$resourceId`?api-version=2021-12-01-preview") -Method PATCH -Payload (ConvertTo-Json $properties)
Step 3: Monitor Storage Growth. The “Storage” meter is a silent budget killer. To monitor the long-term growth of your data lake, you can use Azure Resource Graph queries to track the total size over time, ensuring you set appropriate retention policies per table rather than leaving everything at maximum. A good start is to audit tables with 12-year retention and reduce it to 1-3 years where appropriate based on your compliance needs (e.g., NIS2).
What Undercode Say
- Key Takeaway 1: The new `Plan` field in the `Usage` table is a fundamental building block for cost accountability. For the first time, you can natively track which data plans (Analytics, Basic, Auxiliary) are driving your costs, enabling you to make data-driven decisions about where to store specific log types.
- Key Takeaway 2: Democratizing cost visibility by allowing SOC teams to view the Cost Management blade without Billing Admin permissions is a masterstroke in operational efficiency. It empowers security engineers to manage the financial impact of their data ingestion strategies without relying on a separate, often slower, financial team.
Prediction
In the next 12-18 months, as the March 31, 2027, Azure portal retirement for Sentinel approaches, we will see a massive migration of cost management and security operations into the unified Microsoft Defender portal. This will accelerate the adoption of the Data Lake tier as a cost-saving measure for non-critical logs. Consequently, expect a new wave of third-party tools and open-source scripts focused on granular Data Lake cost optimization and forecast modeling, as manually managing the five new meters will become an indispensable skill for every SOC analyst.
▶️ Related Video (74% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Jeffrey Appel – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
