MirrorMaker

Introduction

What is MirrorMaker? MirrorMaker 2 replicates topics, consumer group offsets, and metadata between Kafka clusters using Kafka Connect.

Understanding the topic

What happens — MirrorMaker:

• Define RPO/RTO and data sovereignty.
• Replicate selected topics cross-cluster.
• Plan failover and conflict handling.
• Capacity-plan for peak + replay.

Visual explanation

Pipeline view:

Consumer polls records from topic partitions
  ↓
Process business logic (DB, API, etc.)
  ↓
Commit offset after success
  ↓
Monitor consumer lag

Step-by-step explanation

1. Subscribe — Consumer joins a group and receives partition assignments.
2. Poll — Fetch records in batches with consumer.poll().
3. Process — Run business logic for each record.
4. Commit — Save offset after successful side effects.
5. Repeat — Continue polling; rebalance if group membership changes.

Informative example

Example:

kafka-topics --bootstrap-server localhost:9092 --create --topic mirrormaker --partitions 6 --replication-factor 3
kafka-console-producer --bootstrap-server localhost:9092 --topic mirrormaker
kafka-console-consumer --bootstrap-server localhost:9092 --topic mirrormaker --from-beginning
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group mirrormaker-service

Execution workflow

Best practices

• Use manual commits for critical side effects.
• Make consumers idempotent with event IDs or unique constraints.
• Keep processing under max.poll.interval.ms or use pause/resume patterns.
• Send poison messages to DLT with failure metadata.

Common mistakes

• Committing before database writes complete.
• Scaling consumers beyond partition count and expecting more throughput.
• Blocking the poll loop with slow downstream calls.

Summary

MirrorMaker — Commit offsets only after successful processing; make handlers idempotent; monitor lag per partition.