Listen to this Post
Introduction:
Microsoft Defender Custom Data Collection is now generally available, fundamentally shifting endpoint security by allowing organizations to collect custom telemetry directly through the existing Microsoft Defender for Endpoint (MDE) sensor without deploying any additional agents. This capability eliminates the need for complex logging infrastructure, enabling defenders to tailor event collection to their specific threat landscape and operational requirements, turning the MDE platform into a highly flexible “Swiss Army knife” for security operations. By defining precise rule-based filters for events like process creation, file writes, and network connections, teams can now fill critical telemetry gaps while maintaining native integration with Microsoft Sentinel for hunting and analysis.
Learning Objectives:
- Master the architecture of custom data collection to expand telemetry beyond default MDE configurations for advanced threat hunting.
- Learn to create and deploy granular data collection rules using both the Microsoft Defender portal and YAML-based rules-as-code methodologies.
- Implement performance monitoring and cost-optimization strategies when integrating custom telemetry into your Microsoft Sentinel workflows.
You Should Know:
1. Understanding Custom Collection Architecture
Custom Data Collection operates as an extension to the existing MDE sensor, allowing security teams to define specific events and conditions for collection through rule-based filtering. This feature addresses a long-standing limitation of MDE: the default telemetry cap of approximately 25,000 events per device per day, which often leads to an incomplete picture of endpoint activity. By defining custom rules, organizations can now ensure that critical events—such as specific PowerShell executions or script behaviors—are always collected and stored in your connected Microsoft Sentinel workspace, alongside default Defender telemetry.
Step‑by‑step guide explaining what this does and how to use it:
– Integration: Custom Collection integrates directly with the MDEClientAnalyzer and leverages the existing MDE sensor infrastructure without needing additional agents.
– Define Rules: Specify event types (e.g., ProcessCreation, FileWrite, NetworkConnection) along with filtering criteria tailored to your threat hunting priorities.
– Deploy: Use the Microsoft Defender portal (Settings > Endpoints > Rules > Custom Data Collection) to define and distribute rules to targeted devices.
– Storage: Collected custom events are stored in Microsoft Sentinel with a “custom” prefix (e.g., DeviceCustomProcessEvents), where they can be queried alongside standard tables.
2. Creating Your First Collection Rule
Effective custom collection begins with well-structured rules that balance comprehensive coverage with performance considerations. For example, if your threat hunting focuses on detecting suspicious script-based attacks, you can create a rule that captures all PowerShell and script-related telemetry events that might otherwise be capped.
Step‑by‑step guide explaining what this does and how to use it:
– Rule to Monitor Suspicious Script Execution (via Defender Portal):
1. Navigate to Settings > Endpoints > Rules > Custom Data Collection.
2. Select Create rule and choose the DeviceCustomScriptEvents table.
3. Under Rule conditions, add filters to narrow the scope: Image contains “powershell.exe”, “pwsh.exe”, or “cscript.exe”.
4. In the Define rule scope, select either “All devices” or target specific dynamic tags (requires pre-configured asset rules).
5. Submit the rule, which typically deploys to targeted endpoints within 20 minutes to one hour.
- Rule to Monitor Suspicious Process Activity (via YAML approach using FalconForce’s toolkit): Create a YAML file named `suspicious_processes.yaml` with the following content:
version: 1.0 rules:</li> <li>name: "Suspicious Process Execution" events:</li> <li>type: ProcessCreation filters:</li> <li>field: Image operator: contains value:</li> <li>"rundll32.exe"</li> <li>"regsvr32.exe"</li> <li>field: CommandLine operator: contains value:</li> <li>"javascript:"</li> <li>"scrobj.dll" collection:</li> <li>field: Image</li> <li>field: CommandLine</li> <li>field: User</li> <li>field: ParentCommandLine
Note: The TelemetryCollectionManager can be used to validate and deploy this YAML.
3. Deploying Rules at Enterprise Scale
Manually creating rules in the Defender portal becomes impractical for large organizations. Microsoft currently lacks a native API for this feature, but third-party open-source tools like FalconForce’s TelemetryCollectionManager (TCM) bridge this gap by providing a rules-as-code approach.
Step‑by‑step guide explaining what this does and how to use it:
1. Download and Install TCM: Clone or download the TCM tool from `https://github.com/FalconForceTeam/TelemetryCollectionManager` and install the associated PowerShell module.
2. Import Module and Initialize Repository:
Install-Module -Name TelemetryCollectionManager -Force Import-Module TelemetryCollectionManager Initialize-TCMRepository -Path "C:\TCM-Rules"
3. Add and Deploy Rules: Place your YAML rules in the repository. Use the TCM CLI to create and update rules in the Defender portal via Azure CLI authentication.
4. Validate Deployment: Use MDE client analyzer to confirm rule activation on targeted devices.
4. Querying Custom Data in Microsoft Sentinel
Once your custom rules are deployed and telemetry starts flowing, the real power of this feature emerges through analysis. All custom-collected data is stored in Sentinel workspaces with a structured schema, allowing you to hunt for threats and build detection rules.
Step‑by‑step guide explaining what this does and how to use it (Linux/Windows commands):
– KQL Query to Verify Rules are Working:
// Count events collected per custom rule on a specific device search in (DeviceCustomFileEvents, DeviceCustomScriptEvents, DeviceCustomNetworkEvents) "your_device_id" | where DeviceId == "your_device_id" | summarize EventCount = count() by RuleName, RuleLastModificationTime, $table
This query helps confirm that rules have reached the endpoint and are actively collecting data.
- KQL Query to Hunt for Suspicious WMI Persistence:
DeviceCustomScriptEvents | where ActionType == "WmiEventConsumerCreated" or ActionType == "WmiEventFilterCreated" | where AdditionalFields.CustomCollection == "true" | project Timestamp, DeviceName, InitiatingProcessFileName, AdditionalFields
This helps identify potential WMI persistence mechanisms that might be missed by standard telemetry.
5. Limitations and Performance Considerations
While powerful, Custom Collection has several important limitations that security engineers must understand to avoid operational issues.
Step‑by‑step guide explaining what this does and how to use it:
– Event Cap: Each custom collection rule is limited to capturing 25,000 events per device within a 24-hour rolling window. If a device reaches this limit, collection for that rule stops until the window resets.
– Deployment Delay: Rule deployment to endpoints typically takes 20 minutes to one hour. Do not expect instant changes.
– License Requirement: A Microsoft Defender for Endpoint P2 license and a connected Microsoft Sentinel workspace are mandatory, with Sentinel ingestion costs applying separately.
– Tag Support: Only dynamic tags (based on asset rules) are supported for device targeting; manual static tags cannot be used directly.
– Cost Monitoring: The feature itself is included with P2 licensing, but the data ingested into Sentinel incurs charges based on your billing arrangement.
6. Performance Optimization and Best Practices
Improperly configured rules can impact endpoint performance. It is crucial to adopt a phased approach and monitor system metrics after deployment.
Step‑by‑step guide explaining what this does and how to use it:
– Start Small: Begin with targeted rules focusing on high-value security use cases rather than broad, all-encompassing collections.
– Monitor Endpoint Performance (Windows/Linux):
– Windows (PowerShell): `Get-Counter “\Process()\% Processor Time” | Sort-Object -Property CookedValue -Descending | Select-Object -First 10`
– Linux: `ps aux –sort=-%cpu | head -10` and `iotop -ao`
– Use Exclusion Filters: Add filters to exclude noise from trusted processes (e.g., `Image` does not contain “C:\Windows\System32\svchost.exe”).
– Establish a Rule Review Process: Regularly review collected data to identify and remove obsolete rules, optimizing for both performance and cost.
What Undercode Say:
- Key Takeaway 1: The General Availability of this feature signals a paradigm shift from MDE as a “black box” EDR to an open, customizable platform where defenders have granular control over visibility, directly addressing the long-standing issue of event capping.
- Key Takeaway 2: The absence of a native public API for rule management is a significant operational hurdle, making community-driven tools like FalconForce’s TelemetryCollectionManager essential for enterprise-scale deployment.
Analysis: By simultaneously releasing open-source management tools and partnering with elite security teams like FalconForce during development, Microsoft has demonstrated a modern approach to vendor-community collaboration. This not only accelerates enterprise adoption but also validates the feature against real-world operational needs. Custom Collection effectively eliminates the trade-off between rich telemetry and endpoint complexity, making it a strategic asset for any mature security operations center (SOC) seeking to reduce blind spots.
Prediction:
Within two years, customizable telemetry collection will become a baseline expectation for all enterprise EDR platforms, forcing competitors to develop similar capabilities. This will foster an ecosystem of specialized, community-driven rule packs targeting specific industries, compliance frameworks (e.g., PCI-DSS, NIST), and threat actor tactics, techniques, and procedures (TTPs). Ultimately, the granularity of this data will enable more accurate machine learning models for anomaly detection, creating a powerful feedback loop where refined telemetry improves detection, which in turn drives better collection strategies.
▶️ Related Video (76% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Jeffrey Appel – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
