Listen to this Post
Introduction:
The integration of Azure Monitor with Fabric Eventhouse and Azure Data Explorer (ADX) marks a pivotal shift in cloud telemetry management. This new pipeline enables the Azure Monitor Agent (AMA) to stream data directly into these analytical engines, bypassing the traditional and often costly Log Analytics workspace. By eliminating manual schema setup through automatic table creation and adjustment, this feature empowers organizations to unify operational telemetry with other business data for powerful, cross-domain analytics at a fraction of the historical cost.
Learning Objectives:
- Configure the Azure Monitor Agent to stream telemetry data directly to a Fabric Eventhouse or Azure Data Explorer cluster.
- Execute advanced Kusto Query Language (KQL) queries to correlate security, performance, and business data in a single pane.
- Implement security hardening for the new data pipeline, including Private Endpoints and managed identities, to protect sensitive log data.
You Should Know:
1. Architecting the New Data Pipeline
The core of this announcement is a direct ingestion pipeline from the Azure Monitor Agent (AMA) to a Kusto-based engine (Eventhouse/ADX). AMA, the successor to the legacy Log Analytics agent, collects data from Azure VMs, Arc-enabled servers, and other sources. Previously, this data was sent to a Log Analytics workspace, which acted as both a storage and query engine. The new capability allows AMA to use a Data Collection Rule (DCR) to bypass the workspace and stream data directly into a table in your Eventhouse or ADX database. This decouples ingestion from analysis, offering greater flexibility and potential cost savings on log storage and ingestion.
Step‑by‑step guide explaining what this does and how to use it.
Step 1: Prerequisites. You must have an Azure subscription with owner permissions, an Azure VM (or Arc-enabled server) with the Azure Monitor Agent installed, and a provisioned Fabric Eventhouse or Azure Data Explorer cluster.
Step 2: Create a Data Collection Rule (DCR). The DCR is the central configuration object that defines what data is collected and where it is sent.
In the Azure Portal, navigate to Monitor > Data Collection Rules > Create.
Associate the DCR with your target Eventhouse or ADX cluster during creation.
In the “Resources” tab, associate the Azure VM(s) that will use this rule.
Step 3: Define Data Sources and Destinations. Within the DCR, specify the performance counters, Windows event logs, or Syslog data you wish to collect. The destination will be your Eventhouse/ADX endpoint. The system will automatically create a table (e.g., AMA_CL) with a schema that matches your data selection.
2. Mastering KQL for Cross-Domain Threat Hunting
With all your operational data flowing into a single, powerful query engine, you can perform correlations that were previously difficult or required complex cross-workspace queries. For security teams, this means you can join VM security logs with application telemetry and network flow data in a single query.
Step‑by‑step guide explaining what this does and how to use it.
Step 1: Access the Query Interface. Navigate to your Fabric Eventhouse or ADX cluster and open the Query editor.
Step 2: Basic Query to Identify Failed Logons. Start with a simple query to find failed Windows authentication events from your VMs. Assuming the table is AMA_CL:
AMA_CL | where EventLog == "Security" | where EventID == 4625 // Windows Logon Failure | project TimeGenerated, Computer, Account_Name, IpAddress
Step 3: Correlate with Network Flows. Imagine you have a separate table `NetworkFlow_CL` from a different source. You can join the data to see if the failed logons came from a suspicious IP that was also scanning your network.
let FailedLogons = AMA_CL | where EventLog == "Security" and EventID == 4625 | project TimeGenerated, Computer, TargetAccount = Account_Name, IpAddress; let SuspiciousIPs = NetworkFlow_CL | where FlowType == "Malicious" // Hypothetical column | project IpAddress; FailedLogons | join kind=inner SuspiciousIPs on IpAddress | summarize FailedAttempts = count() by TargetAccount, IpAddress, Computer
3. Hardening the Data Stream with Network Security
Streaming log data directly to ADX/Eventhouse introduces a new network endpoint that must be secured. To prevent data exfiltration or man-in-the-middle attacks, you must enforce private network connectivity and managed identity authentication.
Step‑by‑step guide explaining what this does and how to use it.
Step 1: Configure a Private Endpoint. Create a Private Endpoint for your Eventhouse or ADX cluster within your Azure Virtual Network (VNet). This ensures that traffic from your VMs to the cluster never traverses the public internet.
Azure CLI Command:
az network private-endpoint create \
--name myAdxPrivateEndpoint \
--resource-group myResourceGroup \
--vnet-name myVnet \
--subnet mySubnet \
--private-connection-resource-id "/subscriptions/{subscription-id}/resourceGroups/{rg}/providers/Microsoft.Kusto/clusters/{adx-cluster-name}" \
--group-id cluster \
--connection-name myConnection
Step 2: Disable Public Network Access. Once the Private Endpoint is configured and validated, modify the cluster’s networking settings to deny all public access. This is a critical step for compliance (e.g., SOC2, HIPAA).
4. Automating Deployment with Infrastructure-as-Code (IaC)
To ensure a consistent and auditable deployment of this monitoring pipeline, define the resources using Infrastructure-as-Code with tools like Bicep or Terraform.
Step‑by‑step guide explaining what this does and how to use it.
Step 1: Create a Bicep Template. Define the ADX cluster, DCR, and association in a `main.bicep` file.
Step 2: Define the ADX Cluster and DCR.
resource adxCluster 'Microsoft.Kusto/clusters@2023-08-15' = {
name: 'adx-cluster-${uniqueString(resourceGroup().id)}'
location: resourceGroup().location
sku: {
name: 'Standard_D11_v2'
tier: 'Standard'
}
}
resource dataCollectionRule 'Microsoft.Insights/dataCollectionRules@2022-06-01' = {
name: 'dcr-to-adx'
location: location
properties: {
dataSources: {
performanceCounters: [
{
name: 'perfCounter'
streams: ['Microsoft-Perf']
samplingFrequencyInSeconds: 60
counterSpecifiers: ['\Processor(_Total)\% Processor Time']
}
]
}
destinations: {
azureDataExplorer: [
{
name: 'adx-destination'
clusterResourceId: adxCluster.id
databaseName: 'myDatabase'
}
]
}
dataFlows: [
{
streams: ['Microsoft-Perf']
destinations: ['adx-destination']
}
]
}
}
Step 3: Deploy. Use `az deployment group create –template-file main.bicep` to deploy the infrastructure.
5. Cost Analysis and Optimization
A primary driver for this new pipeline, as highlighted in the LinkedIn comments, is the high cost of Log Analytics. By streaming to ADX/Eventhouse, you gain more control over storage tiers and compute scaling.
Step‑by‑step guide explaining what this does and how to use it.
Step 1: Estimate Current Log Analytics Costs. Use the Azure Pricing Calculator to understand your current spend on data ingestion and retention in Log Analytics.
Step 2: Compare with ADX Pricing Model. ADX costs are separated into compute (cluster specs) and storage (blob storage). For low-latency querying, you need a running cluster, but you can scale it down or even pause it during off-hours for dev/test environments to save costs.
Step 3: Implement a Data Retention Policy. Use a simple KQL management command to automatically purge old data, something more flexible than Log Analytics’ fixed tiers.
.alter database MyDatabase policy retention softdelete = 30d recoverability = disabled
What Undercode Say:
- Key Takeaway 1: This integration is a strategic move to counter the “Log Analytics tax,” offering a more flexible and potentially cheaper analytical backend for high-volume telemetry, though it requires a higher initial setup and management overhead compared to the fully-managed Log Analytics experience.
- Key Takeaway 2: The automatic schema management is a double-edged sword; it accelerates time-to-value for developers but can introduce challenges for change management and compliance audits where strict, pre-defined schemas are required.
Analysis: The announcement signals Microsoft’s intent to consolidate its data stack around the Kusto engine, positioning Fabric as the unified data fabric for the enterprise. While the initial preview is limited to AMA data (primarily VMs), the product team’s confirmation that Azure service logs are on the roadmap is the most critical piece of information. This will eventually allow organizations to ingest all Azure platform logs (e.g., Activity Logs, Diagnostic Logs from PaaS services) directly into a single, powerful data explorer, fundamentally changing the cloud monitoring and cost management landscape. Security teams should immediately begin evaluating this for their SIEM pipelines, as the query power for threat hunting is significantly enhanced.
Prediction:
Within 18-24 months, direct ingestion to analytical engines like ADX and Eventhouse will become the default pattern for cloud-native logging, relegating traditional log management workspaces to a legacy role. This will fuel the rise of “FinOps for logs,” where organizations actively manage log lifecycle and query costs with the same rigor as their core cloud infrastructure. Furthermore, the fusion of operational and business data in one engine will be a key enabler for AIOps, allowing machine learning models to detect anomalies by correlating application performance with underlying infrastructure health and business KPIs in real-time.
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Vincentlauzon Send – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
