Dead Letter Topics

Introduction

What is Dead Letter Topics? Dead letter topics store poison messages after retries are exhausted so the main consumer can keep processing.

Understanding the topic

What happens — Dead Letter Topics:

• Classify retryable vs poison errors.
• Retry with backoff for transient failures.
• Route poison messages to DLT.
• Alert and replay from DLT after fix.

Visual explanation

Pipeline view:

Consumer polls records from topic partitions
  ↓
Process business logic (DB, API, etc.)
  ↓
Commit offset after success
  ↓
Monitor consumer lag

Step-by-step explanation

1. Subscribe — Consumer joins a group and receives partition assignments.
2. Poll — Fetch records in batches with consumer.poll().
3. Process — Run business logic for each record.
4. Commit — Save offset after successful side effects.
5. Repeat — Continue polling; rebalance if group membership changes.

Informative example

Example:

kafka-topics --bootstrap-server localhost:9092 --create --topic dead-letter-topics --partitions 6 --replication-factor 3
kafka-console-producer --bootstrap-server localhost:9092 --topic dead-letter-topics
kafka-console-consumer --bootstrap-server localhost:9092 --topic dead-letter-topics --from-beginning
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group dead-letter-topics-service

Execution workflow

Best practices

• Use manual commits for critical side effects.
• Make consumers idempotent with event IDs or unique constraints.
• Keep processing under max.poll.interval.ms or use pause/resume patterns.
• Send poison messages to DLT with failure metadata.

Common mistakes

• Committing before database writes complete.
• Scaling consumers beyond partition count and expecting more throughput.
• Blocking the poll loop with slow downstream calls.

Summary

Dead Letter Topics — Commit offsets only after successful processing; make handlers idempotent; monitor lag per partition.