Listen to this Post
Distributed systems are built on retries, replays, and redeliveries. Duplication isn’t rare—it’s inevitable. A wallet getting credited twice, an order being fulfilled again, or an email being resent can break production systems. Designing for idempotency and deduplication is crucial.
🔗 Read the full article here: https://lnkd.in/eet7uVuC
You Should Know:
1. Where Duplication Happens
- Producer Side: Network retries can resend the same message.
- Broker Side: Message queues (Kafka, RabbitMQ) may redeliver unacknowledged messages.
- Consumer Side: Processing delays can lead to duplicate handling.
2. Practical Deduplication Techniques
At the Producer:
- Use unique IDs (UUIDs) for each request.
- Implement idempotency keys in APIs.
Example (Python):
import uuid
request_id = str(uuid.uuid4())
headers = {'Idempotency-Key': request_id}
response = requests.post('https://api.example.com/order', headers=headers, json=data)
At the Broker (Kafka):
- Enable idempotent producers in Kafka:
kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name my_topic --add-config min.insync.replicas=2
At the Consumer:
- Use deduplication tables in databases.
Example (SQL):
CREATE TABLE processed_events (
event_id VARCHAR(255) PRIMARY KEY,
processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Check before processing
INSERT INTO processed_events (event_id) VALUES ('event_123') ON CONFLICT DO NOTHING;
3. Bufstream’s Approach
- Uses deterministic event hashing to detect duplicates.
- Employs TTL-based cleanup to avoid storage bloat.
Example (Redis for Deduplication):
redis-cli SETNX event:123 "processed" EX 86400 Expires in 24h
What Undercode Say
Handling duplicates is a must, not an afterthought. Use:
– Linux commands (awk, sort -u) to filter logs.
– Windows PowerShell to deduplicate files:
Get-Content .\events.log | Sort-Object -Unique > deduped.log
– Cron jobs to clean old deduplication keys:
0 3 redis-cli KEYS "event:" | xargs redis-cli DEL
Always test retry mechanisms in staging before production.
Expected Output:
A resilient system that:
✅ Logs duplicates (`grep “duplicate” /var/log/app.log`)
✅ Automatically discards them (`if redis.get(event_id): return`)
✅ Alerts on suspicious patterns (`Prometheus + Grafana`)
🔗 Further Reading: https://lnkd.in/eet7uVuC
References:
Reported By: Raul Junco – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
