Brokers

Introduction

What is Brokers? Brokers are Kafka servers that store partitions, serve reads/writes, replicate data, and coordinate clients.

Understanding the topic

What happens — Brokers:

• Brokers host topic partitions on disk.
• One broker is leader per partition.
• Followers replicate for fault tolerance.
• Controller broker manages cluster metadata.

Visual explanation

Pipeline view:

Producer
  ↓
Topic partition log
  ↓
Consumer group A
  ↓
Consumer group B
Replay is possible because records are retained

Step-by-step explanation

1. Start broker — Load logs and join cluster.
2. Lead partitions — Serve produce/fetch for assigned leaders.
3. Replicate — Followers fetch from leaders.
4. Fail over — New leader elected on failure.

Informative example

Example:

kafka-topics --bootstrap-server localhost:9092 --create --topic brokers --partitions 6 --replication-factor 3
kafka-console-producer --bootstrap-server localhost:9092 --topic brokers
kafka-console-consumer --bootstrap-server localhost:9092 --topic brokers --from-beginning
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group brokers-service

Execution workflow

Best practices

• Design events around business facts: OrderCreated, PaymentAuthorized, InventoryReserved.
• Pick keys based on ordering and load distribution.
• Document owner, retention, schema, partitions, and consumers for every topic.
• Assume duplicate delivery and build idempotent consumers early.

Common mistakes

• Expecting global ordering across a topic.
• Creating one partition because local demos work.
• Skipping schema governance until multiple teams deploy independently.

Summary

Brokers — Start every design with event ownership, topic naming, key choice, schema contract, retention, replay plan, and monitoring.