Listen to this Post
Introduction:
Detection Engineering CI/CD pipelines often focus on automating unit tests, linting, and back-testing, but the backlog of missing detections and tuning opportunities remains largely manual. By integrating Anthropic’s Claude AI into GitHub Actions, you can perform automated gap analysis between your local detection repository and open‑source rule sets like Sigma, generating a prioritized coverage report that aligns with your log sources and threat model.
Learning Objectives:
- Implement a GitHub Action that uses Claude AI to compare your detection rules against Sigma’s latest release
- Generate a coverage report that highlights missing Sigma rules relevant to your environment
- Automate backlog creation by turning AI‑identified gaps into actionable tickets in Jira or ServiceNow
You Should Know:
1. Setting Up Your Detection Engineering Lab Repository
Start with a GitHub repository structured for Sigma‑based detection rules. Include folders for rules/, logsources/, and config/. Use the Sigma CLI to validate rule syntax.
Linux / macOS setup:
Install sigma-cli
pip install sigma-cli
Initialize repo structure
mkdir -p detection-lab/{rules/windows,logsources,config}
cd detection-lab
git init
echo " Detection Lab" > README.md
Windows (PowerShell):
pip install sigma-cli New-Item -Path "detection-lab\rules\windows", "detection-lab\logsources", "detection-lab\config" -ItemType Directory
Step‑by‑step: This folder structure allows the GitHub Action to index your custom rules. The `logsources` folder defines what telemetry you ingest (e.g., Sysmon, Windows Event IDs). The `config` folder holds transformation settings for Sigma output.
- Building the GitHub Action Workflow with Claude API
Create.github/workflows/sigma-gap-analysis.yml. The workflow runs on schedule or push, fetches the latest Sigma release, and invokes Claude via Python.
name: Sigma Gap Analysis with Claude
on:
schedule:
- cron: '0 0 1' Weekly every Monday
workflow_dispatch:
jobs:
gap-analysis:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: pip install anthropic pyyaml requests
- name: Clone SigmaHQ repo (latest tag)
run: |
git clone --depth 1 --branch master https://github.com/SigmaHQ/sigma.git sigma-repo
- name: Run gap analysis script
env:
CLAUDE_API_KEY: ${{ secrets.CLAUDE_API_KEY }}
run: python gap_analysis.py
- name: Upload coverage report
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage_report.md
Step‑by‑step: The action checks out your detection repo, clones the official Sigma repository, runs a Python script that uses Claude to compare rule coverage, and saves a markdown report as an artifact.
- Performing Sigma Rule Gap Analysis via Python Script
Create `gap_analysis.py` in your repo root. This script enumerates Sigma rules (grouped by category, e.g.,process_creation,file_event) and your local rules, then asks Claude to identify missing rules that fit your log sources and threat model.
import os, yaml, glob, json
from anthropic import Anthropic
client = Anthropic(api_key=os.environ["CLAUDE_API_KEY"])
def get_sigma_rules_by_category(path="sigma-repo/rules"):
categories = {}
for file in glob.glob(f"{path}//.yml", recursive=True):
with open(file) as f:
data = yaml.safe_load(f)
cat = data.get("logsource", {}).get("category", "other")
categories.setdefault(cat, []).append({
"title": data.get("title"),
"id": data.get("id"),
"logsource": data.get("logsource")
})
return categories
def get_local_rules_summary(local_path="rules"):
local_ids = set()
for file in glob.glob(f"{local_path}//.yml", recursive=True):
with open(file) as f:
data = yaml.safe_load(f)
local_ids.add(data.get("id"))
return local_ids
sigma_rules = get_sigma_rules_by_category()
local_ids = get_local_rules_summary()
prompt = f"""You are a detection engineer. Our log sources: Windows Event IDs 4688, 4104, Sysmon 1/3/10. Threat model includes ransomware, credential dumping, lateral movement.
We already have {len(local_ids)} custom rules. Below are Sigma rule categories with top 10 rule IDs and titles per category.
{json.dumps({cat: [r['id'] for r in rules[:10]] for cat, rules in sigma_rules.items()}, indent=2)}
List the top 5 missing Sigma rules (by ID and title) that best fit our environment. Explain why each is critical.
"""
response = client.messages.create(
model="claude-3-haiku-20240307",
max_tokens=1000,
messages=[{"role": "user", "content": prompt}]
)
with open("coverage_report.md", "w") as f:
f.write(" Sigma Gap Analysis Report\n\n")
f.write(response.content[bash].text)
Step‑by‑step: This script parses both rule sets, creates a prompt that provides Claude with your log sources and threat model, then writes Claude’s recommendations to a markdown report. You can expand it to include direct comparisons of detection logic using embeddings.
4. Generating a Coverage Report and Actionable Guidance
The output `coverage_report.md` contains Claude’s prioritized list of missing rules. Enhance the script to map each missing rule to a mitigation technique (e.g., MITRE ATT&CK). Example of an enhanced output:
Sigma Gap Analysis Report Generated: 2025-03-15 Environment: Windows endpoints + Sysmon High‑Priority Missing Rules <table> <thead> <tr> <th>Sigma ID</th> <th></th> <th>MITRE Tactic</th> <th>Log Source</th> </tr> </thead> <tbody> <tr> <td>S-1234</td> <td>Suspicious LSASS Access</td> <td>Credential Access</td> <td>Sysmon Event ID 10</td> </tr> <tr> <td>S-5678</td> <td>BloodHound Ingestor Execution</td> <td>Discovery</td> <td>Process Creation 4688</td> </tr> </tbody> </table> Implementation Guidance - S-1234: Add rule targeting `EventID 10` with `TargetImage` containing <code>lsass.exe</code>. Use sigma-cli to convert to Splunk/Elastic. - S-5678: Combine with parent process `powershell.exe` and command‑line containing <code>Sharphound</code>.
Step‑by‑step: The report becomes your backlog source. Each row can be turned into a Jira ticket via the next step.
5. Automating Backlog Creation in Jira or ServiceNow
Extend the GitHub Action to call Jira’s REST API for each missing rule. Add this Python snippet after generating Claude’s response:
import requests
from requests.auth import HTTPBasicAuth
jira_url = "https://your-domain.atlassian.net/rest/api/3/issue"
auth = HTTPBasicAuth(os.environ["JIRA_EMAIL"], os.environ["JIRA_API_TOKEN"])
for missing in parsed_missing_rules: parsed from Claude response
payload = {
"fields": {
"project": {"key": "SEC"},
"summary": f"[Detection Gap] {missing['title']}",
"description": f"Missing Sigma rule {missing['id']}.\nGuidance: {missing['reason']}",
"issuetype": {"name": "Task"}
}
}
requests.post(jira_url, json=payload, auth=auth)
Step‑by‑step: This closes the loop from AI analysis to actionable work. Store Jira credentials in GitHub Secrets (JIRA_EMAIL, JIRA_API_TOKEN). Optionally add labels like `detection-backlog` and sigma-gap.
- Integrating Linting and Back‑Testing into the Same Pipeline
Before running the gap analysis, validate your existing rules to avoid false positives. Add these steps to your workflow:
- name: Lint Sigma rules with sigma-cli run: | sigma check rules/ --verbose - name: Back-test against sample logs (optional) run: | sigma convert -t splunk rules/ -o /tmp/out Run a mock log through Splunk search (not shown for brevity)
Step‑by‑step: Linting catches YAML syntax errors and missing required fields. Back‑testing ensures that your current rules do not produce too many false alerts. Use `pytest` with a small corpus of known benign and malicious events to validate detection logic.
- Hardening Your CI/CD Pipeline for AI Secrets and API Security
Claude API keys and Jira tokens are critical. Follow these best practices:
- GitHub Secrets: Never hardcode keys. Use
secrets.CLAUDE_API_KEY. - OIDC for Jira: Replace static tokens with OpenID Connect (OIDC) to avoid secret rotation.
- Rate limiting: Claude has token/min limits. Add `time.sleep(1)` between API calls if analysing many rules.
- Output scanning: Use GitHub’s `advanced-security` to scan the generated report for accidental secret exposure.
- Linux command to verify no secrets in logs: `grep -r “sk-ant-api” ./` inside your workflow after execution.
Step‑by‑step: Add a post‑job step that runs `trufflehog` on the artifact directory to prevent leaking API keys into report files.
What Undercode Say:
- Key Takeaway 1: Most SOCs ignore backlog automation — this GitHub Action shifts detection engineering from reactive to proactive by letting AI find blind spots against community‑driven Sigma rules.
- Key Takeaway 2: Claude’s natural language understanding can interpret your log sources and threat model better than simple keyword matching, producing context‑aware recommendations instead of raw rule comparisons.
Analysis: Michael L’s approach solves a real pain point: detection teams drown in manual gap assessments. By embedding Claude into a CI/CD trigger, the backlog becomes a living artifact updated weekly. The script is extensible — you could swap Sigma for Splunk ES, Elastic Detection Rules, or even internal threat intel feeds. The 10‑line core script can be expanded with vector embeddings to match rule semantics, not just IDs. However, caution is needed: Claude may hallucinate rule names; always verify suggestions against your data sources. This pattern also encourages a “detection as code” culture where AI becomes a collaborative peer reviewer.
Prediction:
Within 18 months, autonomous detection engineering pipelines will not only identify gaps but also generate draft Sigma rules using fine‑tuned LLMs, complete with unit tests. SOC analysts will shift from writing rules to validating AI‑generated detections, reducing average detection backlog time from weeks to hours. As cloud logs (AWS CloudTrail, Azure Activity) become more standardised, cross‑environment gap analysis will merge with CSPM tools, enabling real‑time coverage scoring — and attackers will need to evade both human and machine‑generated detection logic simultaneously.
▶️ Related Video (82% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Ml2025 When – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
