Log Compaction

Introduction

What is Log Compaction? Log compaction keeps the latest value per key, ideal for changelog topics and current-state streams.

Understanding the topic

What happens — Log Compaction:

• Plan partitions and replication factor.
• Configure retention and compaction.
• Monitor ISR, lag, disk.
• Scale brokers and partitions with load.

Visual explanation

Pipeline view:

Input topic
  ↓
filter/map/join/window
  ↓
state store + changelog
  ↓
output topic

Step-by-step explanation

1. Create/configure — kafka-topics or API.
2. Replicate — Followers stay in ISR.
3. Observe — Metrics and alerts.
4. Scale — Brokers/partitions/consumers.

Informative example

Example:

kafka-topics --bootstrap-server localhost:9092 --create --topic log-compaction --partitions 6 --replication-factor 3
kafka-console-producer --bootstrap-server localhost:9092 --topic log-compaction
kafka-console-consumer --bootstrap-server localhost:9092 --topic log-compaction --from-beginning
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group log-compaction-service

Execution workflow

Best practices

• Choose keys before joins and aggregations.
• Name internal topics and state stores clearly.
• Test topology with realistic ordering and late events.
• Monitor processing latency, commit rate, and restore time.

Common mistakes

• Unexpected repartition topics from wrong key choices.
• Ignoring late-arriving events in windows.
• Underestimating state-store disk and restore time.

Summary

Log Compaction — Design topology around keys, state stores, window boundaries, changelog topics, and exactly-once needs.