Listen to this Post
Introduction:
In the ever-evolving landscape of Security Information and Event Management (SIEM), the ability to ingest data from custom or proprietary sources quickly and efficiently is paramount. Traditional methods often involve maintaining costly and complex middle-tier infrastructure (like VMs or Function Apps) just to parse and forward logs. Microsoft Sentinel’s Codeless Connector Framework (CCF) revolutionizes this process by allowing security engineers to configure a fully managed, cloud-native pipeline directly through the REST API. This article dissects the architecture of a Push Codeless Connector, moving beyond theoretical concepts to provide a practical, step-by-step guide on how these connectors are built and validated using the Azure Resource Manager (ARM) framework.
Learning Objectives:
- Understand the core architecture of a Push Codeless Connector, including the roles of Data Collection Endpoints (DCE) and Data Collection Rules (DCR).
- Differentiate between traditional ingestion methods and the Codeless Connector Framework (CCF) approach.
- Learn how to define and deploy a connector configuration using Infrastructure as Code (IaC).
- Master the validation strategy across the control plane, data plane, and user interface.
- Acquire practical commands and steps to test and verify data ingestion into custom tables.
You Should Know:
- Deconstructing the Architecture: DCE, DCR, and Custom Tables
Before writing any code, you must understand the underlying Azure infrastructure that makes the Push connector work. It is not a single resource but a combination of three critical components configured to work in harmony.
- Data Collection Endpoint (DCE): This is the ingestion URL endpoint. It acts as the front door for your logs. Instead of sending data to a generic Log Analytics endpoint, you send it to a unique DCE.
- Data Collection Rule (DCR): This is the brain of the operation. It defines the structure of the incoming data (the stream), transforms it (using KQL transformations), and defines the destination (the specific table in the Log Analytics workspace).
- Custom Table: While you can ingest into default tables, the CCF shines when creating custom tables tailored to your specific data schema.
Step‑by‑step guide: Creating the infrastructure foundation
To emulate the setup, you would typically start by creating these resources using the Azure CLI or PowerShell.
Azure CLI Commands (Linux/macOS):
Log in to Azure
az login
Create a Resource Group (if not exists)
az group create --name SentinelConnectorRG --location eastus
<ol>
<li>Create a Data Collection Endpoint
az monitor data-collection endpoint create \
--name "MyPushConnector-DCE" \
--resource-group SentinelConnectorRG \
--location eastus \
--public-network-access "Enabled"</p></li>
<li><p>Note the logsIngestion URI from the output
Example: "logsIngestion": {"endpoint": "https://mypushconnector-dce-xxxx.eastus-1.ingest.monitor.azure.com"}
PowerShell (Windows):
Connect-AzAccount New-AzResourceGroup -Name "SentinelConnectorRG" -Location "EastUS" Create DCE $dce = New-AzDataCollectionEndpoint -ResourceGroupName "SentinelConnectorRG" ` -Name "MyPushConnector-DCE" ` -Location "EastUS" ` -PublicNetworkAccess "Enabled" Write-Host $dce.LogsIngestion[bash].Endpoint
- Defining the Stream and DCR with ARM Templates
The “codeless” aspect is managed via the `Microsoft.SecurityInsights/settings` resource type, specifically the `CodelessUiConnector` or `CodelessApiConnector` kind. However, the underlying Push mechanism relies on a specific DCR definition. You must define the transformation logic that maps your raw JSON logs to columns in your custom table.
Step‑by‑step guide: Writing the DCR transformation
The DCR includes a `streamDeclarations` section defining the columns and a `transformKql` section that processes the incoming data. Assume your raw data has `source_ip` and `event_type` fields.
DCR JSON Snippet (Example):
{
"properties": {
"dataCollectionEndpointId": "/subscriptions/.../providers/.../myPushConnector-DCE",
"streamDeclarations": {
"Custom-MyAppStream": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "SourceIP",
"type": "string"
},
{
"name": "EventType",
"type": "string"
}
]
}
},
"dataFlows": [
{
"streams": [
"Custom-MyAppStream"
],
"destinations": [
"myCustomTable"
],
"transformKql": "source | project TimeGenerated=todatetime(Time), SourceIP=source_ip, EventType=event_type"
}
]
}
}
To deploy this DCR via Azure CLI:
az monitor data-collection rule create --name "MyAppDCR" \ --resource-group "SentinelConnectorRG" \ --location "eastus" \ --rule-file "dcr-definition.json"
3. The Codeless Connector Configuration in ARM
The connector itself (the UI component visible in the Sentinel Data Connectors blade) is defined separately. This configuration tells Sentinel about your DCE and DCR. It links the visual connector blade to the backend ingestion infrastructure.
Step‑by‑step guide: Connecting the UI to the Data Pipeline
You need to create a CodelessApiConnector. This ARM template points to the DCE and DCR and provides the instructions that populate the connector pane in the Azure portal.
ARM Template Deployment (PowerShell):
Deploy the Codeless Connector definition New-AzResourceGroupDeployment -ResourceGroupName SentinelConnectorRG ` -TemplateFile "codeless-connector.json" ` -TemplateParameterFile "codeless-connector.parameters.json"
Inside the parameters file, you would specify:
connectorUiConfig: The title, instructions, and publisher details.instructionSteps: The copyable commands or scripts users will see to start sending data (often including the DCE URL and the DCR immutable ID).
- Validation Strategy: Control Plane, Data Plane, and UX
Once deployed, a robust validation strategy is required to ensure the connector works. This involves three layers:
– Control Plane: Did the ARM deployment succeed? Is the DCR linked correctly?
– Data Plane: Can we actually send data to the endpoint?
– UX: Does the connector show “Connected” in the portal?
Step‑by‑step guide: Sending a test log to the Data Plane
To test the ingestion pipeline independently of the UI, you use the `Ingestion` REST API against your DCE. You will need the DCE endpoint, the DCR Immutable ID, and the stream name.
PowerShell Script to Send Test Data:
Variables
$dceEndpoint = "https://mypushconnector-dce-xxxx.eastus-1.ingest.monitor.azure.com"
$dcrImmutableId = "dcr-abcdefghijk1234567890" Get this from the DCR overview page
$streamName = "Custom-MyAppStream"
Construct the URL
$ingestUrl = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/$streamName`?api-version=2023-01-01"
Create a test log entry
$body = @(
@{
Time = (Get-Date -Format "yyyy-MM-ddTHH:mm:ss.fffZ")
source_ip = "192.168.1.50"
event_type = "FailedLogin"
}
) | ConvertTo-Json -Compress
Send the data (using Managed Identity or Entra ID Token)
$accessToken = (Get-AzAccessToken -ResourceUrl "https://monitor.azure.com").Token
$headers = @{
"Authorization" = "Bearer $accessToken"
"Content-Type" = "application/json"
}
Invoke-RestMethod -Uri $ingestUrl -Method Post -Body $body -Headers $headers -ContentType "application/json"
Write-Host "Test data sent. Check your Sentinel logs in 5-10 minutes."
5. Production Readiness: Security and Hardening
In a production environment, you cannot rely on generic authentication. The Push Codeless Connector must be hardened.
– Managed Identity: The DCE should only accept tokens from specific Managed Identities (e.g., the identity of a specific application or VM sending logs).
– Network Security: DCEs support Private Link. If your logs originate from a VNet, you should disable public network access and configure Private Endpoints.
Step‑by‑step guide: Restricting Access to the DCE
To harden the endpoint, update the DCE to allow access only from a specific Virtual Network.
Azure CLI Command to update network ACLs (conceptual):
This command updates the network ACL to block public access and allow a specific subnet
az monitor data-collection endpoint update \
--name "MyPushConnector-DCE" \
--resource-group "SentinelConnectorRG" \
--public-network-access "Disabled" \
--network-acls "{\"publicNetworkAccess\":\"Disabled\",\"allowedLinks\":[{\"category\":\"VirtualNetwork\",\"virtualNetworkResourceId\":\"/subscriptions/.../virtualNetworks/MySecVnet\"}]}"
What Undercode Say:
- Infrastructure as Code is Non-Negotiable: The complexity of managing DCEs, DCRs, and connectors manually is too high. Adopting ARM/Bicep for CCF ensures consistency, version control, and the ability to roll back changes. This is the DevSecOps model applied to SIEM ingestion.
- Shift Left on Data Transformation: The CCF pushes data transformation logic from a middleware script to the ingestion pipeline (DCR). This significantly reduces the attack surface and operational overhead. It transforms the security engineer’s role from maintaining servers to writing and optimizing KQL transformations.
Analysis: The move towards codeless connectors represents a broader industry shift toward “Security as Code.” By leveraging cloud-native components like DCE and DCR, organizations can ingest data at scale without the fragility of custom-built parsers. However, this requires a mindset shift: security teams must now become proficient in Azure Resource Manager and Kusto Query Language (KQL) for data transformation, rather than just Python or PowerShell scripting. The validation strategy highlighted—separating control, data, and UX planes—is a best practice borrowed from software development that drastically reduces the time to confidently deploy a new log source.
Prediction:
We will see a decline in custom-built, agent-based log shippers hosted on virtual machines within the next 18-24 months. The Codeless Connector Framework will evolve to support not just push ingestion but also pull-based logic (e.g., polling APIs) natively within the Azure management plane. This will force third-party SIEM competitors and open-source log management tools to either offer similar “serverless ingestion” capabilities or risk being seen as legacy, high-maintenance alternatives. Microsoft Sentinel is effectively commoditizing the data ingestion layer, making the differentiator the quality of analytics and threat hunting, not the method of log collection.
▶️ Related Video (80% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Uros Babic – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
