Listen to this Post
Managing logs across multiple AWS accounts can be challenging, but AWS CloudWatch, S3, and Kinesis offer a robust solution. Marcel Kennert provides a detailed implementation using AWS Cloud Development Kit (CDK), complete with a GitHub repository for hands-on practice.
Source: How to Build a Centralized Logging System from Scratch with AWS CDK
You Should Know:
Key AWS Services for Centralized Logging
- Amazon CloudWatch Logs – Collects and stores logs from AWS services.
- Amazon S3 – Acts as a durable storage for log archives.
3. Amazon Kinesis – Enables real-time log processing.
- AWS CDK – Infrastructure-as-code (IaC) tool to automate deployment.
Step-by-Step Implementation
1. Set Up AWS CDK
Ensure AWS CDK is installed and configured:
npm install -g aws-cdk cdk bootstrap aws://ACCOUNT-NUMBER/REGION
2. Clone the GitHub Repository
git clone https://github.com/[bash].git cd centralized-logging-cdk
3. Deploy the Stack
cdk deploy --all
4. Configure Cross-Account Logging
Modify `lib/centralized-logging-stack.ts` to allow log ingestion from multiple accounts:
new logs.CrossAccountDestination(this, 'CrossAccountLogDestination', {
targetArn: 'arn:aws:logs:REGION:ACCOUNT_ID:destination:LOG_DESTINATION_NAME',
roleName: 'CrossAccountLogRole'
});
5. Enable Kinesis Data Stream for Real-Time Processing
aws kinesis create-stream --stream-name LogStream --shard-count 1
6. Automate Log Export to S3
Use AWS Lambda to process and export logs:
import boto3
def lambda_handler(event, context):
s3 = boto3.client('s3')
s3.put_object(Bucket='log-bucket', Key='logs/exported-log.json', Body=event['logData'])
What Undercode Say
Centralized logging in AWS is essential for security, compliance, and debugging. Leveraging AWS CDK simplifies deployment, while Kinesis and S3 ensure scalability. Below are additional Linux and Windows commands to enhance log management:
Linux Log Management Commands
View system logs journalctl -xe Follow live logs tail -f /var/log/syslog Search logs for errors grep -i "error" /var/log/nginx/error.log Rotate logs manually logrotate -f /etc/logrotate.conf
Windows Log Management Commands
Get recent system events Get-EventLog -LogName System -Newest 10 Export logs to CSV Get-WinEvent -LogName "Application" | Export-Csv "app_logs.csv" Filter specific logs Get-EventLog -LogName Security -InstanceId 4624
AWS CLI Logging Commands
List CloudWatch log groups aws logs describe-log-groups Export logs to S3 aws logs create-export-task --task-name "ExportTask" --log-group-name "/aws/lambda/my-function" --from 1625097600000 --to 1625184000000 --destination "my-log-bucket"
Expected Output: A fully automated, cross-account centralized logging system with real-time processing capabilities.
For further reading, check the original guide: AWS Centralized Logging with CDK.
References:
Reported By: Darryl Ruggles – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
