Apache Kafka Architecture

Data Engineering & Modern Data Platforms · Lesson 37 · Real-Time Data Engineering and Kafka

Lesson Overview

The contract from Streaming Data Architecture names two topics, learning.progress and learning.enrollment, the keys they use, and the cursors each product owns. A platform ticket came back the same day asking for "a Kafka cluster big enough for real time," with a diagram that showed one box labeled Broker and an arrow into a warehouse. That diagram cannot answer the questions the landing job will ask at 02:00. Which process accepted the write? Which process can still serve it if that machine dies? Where is the offset stored? Who is allowed to change the topic? And why did a notebook's bootstrap.servers list succeed against one address and then fail when that address was a controller-only listener?

Apache Kafka, as this course runs it, is a cluster of brokers that store partition logs, plus a controller quorum that stores the metadata those brokers serve. Current Apache Kafka uses KRaft for that quorum. Kafka 4.0 removed ZooKeeper. If you meet ZooKeeper in an older runbook, treat it as a migration debt and read ZooKeeper vs KRaft before you touch it. This lesson does not operate a migration. It teaches the data path and the metadata path well enough that a data engineer can place the progress bus on a cluster, read a describe output, and refuse a design that stores the only copy of an acknowledged event on a single disk.

The specialty primer's Kafka Architecture and Brokers pages go further into process layout, listener names, and operational drills. Use them when you are on the platform rotation. This lesson stays on the pipeline: produce and fetch for learning.progress, what a partition leader is, what replication is for, and which failures the landing job can survive without changing its cursor. Partition counts, retention, and min.insync.replicas are specified in Kafka Topics, Partitions and Replication. Producer calls are the next lesson. The architecture has to exist before those settings mean anything.

Learning Objectives

By the end of this lesson you will be able to:

1. Describe a Kafka cluster as brokers serving partition logs and a KRaft controller quorum serving metadata.
2. Explain a topic as a named set of partitions, and a partition as an ordered, append-only log with offsets.
3. Trace a produce and a fetch to the leader replica of a partition, and say what followers are for.
4. Read an Admin describe result and map leaders, replicas, and in-sync replicas to the progress topic.
5. Separate the data plane from the metadata plane, including what bootstrap.servers is for.
6. State the durability requirement the bus inherited from lesson 36, and which single-node picture violates it.
7. Assign ownership: platform owns the quorum and the disks; data engineering owns the topic contract and the consumer cursors.

Prerequisites

• Streaming Data Architecture: topic names, keys, event time, and the consumer matrix. This lesson does not reopen them.
• Relational Database Internals: a write-ahead log is already a familiar append-only structure. A Kafka partition is that idea used as the integration surface, not as PostgreSQL's crash-recovery log.
• Data Formats and Serialization: record values are bytes. The broker does not parse your JSON.
• /learn/kafka/architecture, /learn/kafka/brokers, and /learn/kafka/zookeeper-vs-kraft if you want the operator's tour in parallel. The Java types used here are from org.apache.kafka.clients.admin in Apache Kafka's own client, artifact kafka-clients.

Detailed Core Concepts

Brokers store logs

A broker is a Kafka server process that stores data and serves client requests. On disk, each replica of a partition is a directory of log segments under log.dirs. The directory holds the active segment the broker appends to, older segments that have rolled, and index files that map offsets and timestamps to positions in the segment. You do not query those files with SQL. You append records and you fetch them by offset.

For the progress bus, a broker is not "the streaming server" in the abstract. It is one copy of some of the partitions of learning.progress and learning.enrollment. If you have three brokers and a replication factor of three, each partition has a replica on every broker. If you have three brokers and a replication factor of one, each partition has a single copy, and the durability sentence from lesson 36 is already false. Replication factor is a topic property. Lesson 39 sets it. The architecture only insists that you can see, from a describe call, that more than one broker is listed for each partition you acknowledge writes to.

Brokers are replaceable disks with a process in front of them, as long as another in-sync replica exists. They are not replaceable if the topic was created with one replica, or if all replicas lived on one volume because log.dirs pointed every broker at the same network disk. Shared storage that fails as a unit is one failure domain wearing three hostnames.

Controllers store metadata

Something has to know that learning.progress exists, that it has a given number of partitions, which broker is the leader of partition 4, and which configuration the topic carries. That something is the controller quorum. In KRaft mode the controllers are Kafka processes with process.roles including controller. They replicate a metadata log among themselves using the KRaft protocol, a Raft quorum. Brokers replicate the metadata they need from that quorum. Clients do not write progress events to the controller. They write them to the leader broker of a partition.

A combined process (process.roles=broker,controller) runs both roles in one JVM. That is a reasonable shape for a small bus: three machines, each a broker and a controller, so you can lose one machine and still have a metadata quorum and a data replica. Dedicated controllers are the shape a platform team uses when a metadata stall must not sit on the same heap as a large fetch, or when the data plane scales independently. TechLearningPro's bus can live on either shape. The data engineer records which one the platform provided, because the failure story differs. Losing a dedicated controller does not remove a replica of partition 4. Losing a combined node removes both a controller vote and a replica.

Kafka 4.0 clusters do not have a ZooKeeper ensemble beside them. Do not add one because an old diagram still has it. Do not point bootstrap.servers at a ZooKeeper port. Clients bootstrap against broker listeners.

Three controllers tolerate the loss of one and still elect a leader for the metadata log. Two controllers do not give you that tolerance: a single loss can stop metadata updates, which means you cannot create topics, elect leaders, or rely on a clean failure response. A one-controller lab is fine for a laptop and dishonest as a production diagram for a bus that instructors use during a live lesson.

Topics, partitions, and offsets

A topic is a name and a configuration, plus an ordered set of partitions. learning.progress is the name in the contract. The partitions are the physical logs. Partition 0 and partition 1 do not share an order. Within one partition, offsets are a contiguous sequence of integers assigned by the leader as it appends: 0, then 1, then 2. The offset is the position of that record in that partition for the life of the log. It is not a global id across the topic. The landing manifest in lesson 36 therefore names topic, partition, and offset together. An offset alone is ambiguous.

The producer chooses a partition. With a key, the default partitioner hashes the key and selects a partition. The same learner_id stays on the same partition for a stable partition count, which is the per-learner order the contract asked for. Without a key, current Kafka producers use a sticky partitioner that fills a batch to one partition and then moves, which is a throughput behavior, not an ordering promise you should use for progress. Lesson 38 shows the client. Lesson 39 decides how many partitions exist. This lesson's point is smaller: the topic is not one file, and "the offset" is not one number.

Adding a partition is a metadata change the controller commits. It does not reshuffle old records. It does change where new keys land if you use the default hash, because the modulus changed. That consequence is lesson 39. Mentioning it here stops an architecture review from treating partition count as a purely operational knob with no data effect.

Leaders, followers, and the in-sync set

Each partition has one leader replica and zero or more followers. Producers send writes to the leader. Followers fetch from the leader and append to their own logs. Consumers, by default, also fetch from the leader. The leader is the only replica that assigns offsets for new records.

The in-sync replica set, the ISR, is the leader plus the followers that have kept up within the time the cluster allows (replica.lag.time.max.ms on the broker). A follower that stops fetching drops out of the ISR. It is still a replica. It is not a replica the leader will wait for when the producer asks for a fully replicated acknowledgement. When the follower catches up, it can rejoin.

If the leader fails, the controller elects a new leader from the ISR, provided the cluster is configured to refuse an out-of-date replica. That refusal is unclean.leader.election.enable=false, the safe default you want for this bus. Electing a stale replica can serve data that acknowledged writes had already passed, which looks like the log going backwards. Availability that rewinds offsets will corrupt the landing manifest. Lesson 39 sets the durability triple: replication factor, minimum ISR, and producer acknowledgements. The architecture names the roles so those settings have a place to land.

A diagram with a leader and no followers is a single copy. Call it a lab. Do not call it the progress bus.

The record the broker actually stores

The broker stores a record batch. Each record has an offset, a timestamp, a key, a value, and optional headers. The timestamp is either the producer's timestamp (CreateTime) or the time the leader appended (LogAppendTime), controlled by the topic's message.timestamp.type. Lesson 36 put analytics on the payload's event_time precisely so this broker timestamp can stay an operational column. Headers are not part of the key. A later compaction policy, if you enable one on the enrollment topic, keeps the latest record per key and does not treat a header as identity.

The value is bytes. Kafka will compress a batch if the producer asks, and it will reject a batch that exceeds the configured maximum size. It will not validate progress_pct. The validator from lesson 36 runs in the producer tests and in the landing consumer. Putting validation "in Kafka" without a component that runs your code is a sentence that means nothing. Schema Registry, lesson 43, can reject a payload that fails a registered schema at the client. It is still not the broker parsing business rules.

Clients, bootstrap, and metadata cache

A client is given bootstrap.servers, a list of broker addresses that speak the client listener. The client connects to one, fetches cluster metadata, and then sends each produce or fetch to the leader of the relevant partition. The list is a bootstrap, not a directory of every broker you must always reach, and not a list of controllers. If every address in the list is down, the client cannot start. If one is up, the client can discover the rest. That is why a single load balancer in front of one broker is a weaker bootstrap than two or three broker addresses: you have reintroduced a single process in the only path that can hand out metadata.

The client caches metadata and refreshes it on a timer and on errors such as an unknown leader. A leadership change is a brief failure at the client, then a refresh, then a retry against the new leader. Idempotent producer retries, which lesson 41 scopes carefully, exist so that this refresh does not become a duplicate business fact by itself. The architecture's job is to expect leader changes as normal, not as data loss.

Admin, in org.apache.kafka.clients.admin, is the supported client for describing the cluster and topics. Shell scripts that ship with the Apache Kafka distribution (kafka-topics.sh, kafka-metadata.sh) talk to the same protocol. A dashboard that scrapes JMX is an operator view. The describe call is the one a pipeline review can paste into a ticket.

Internal topics you will see and should not repurpose

__cluster_metadata is the KRaft metadata log. __consumer_offsets stores committed consumer positions. Both are topics with partitions and replicas, because Kafka uses its own log for its own coordination. They are not landing zones for progress events. A producer ACL that can write internal topics is a misconfiguration. Lesson 40 explains how group offsets use __consumer_offsets. Until then, if a disk report shows that topic growing, the cause is group commit traffic and compaction lag, not your JSON.

What this cluster is not

It is not the lakehouse. It is not the cohort table. It is not a query engine. Consumers that need SQL read the raw zone or the warehouse after landing. It is not a cross-region product by default. One cluster in one region is the assumption for this bus. Mirroring to another region is MirrorMaker in the primer, and it is a new cursor and a new lag budget, not a checkbox on the topic.

It is also not infinitely ordered. The total order of the topic does not exist. The order of each partition does. Any architecture slide that draws one arrow labeled "ordered stream" under a twelve-partition topic is the slide you correct before it becomes a dashboard requirement.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph clients [Clients of the progress bus]
        player[Player producer]
        landing[Landing consumer]
        tile[Spark Structured Streaming]
    end

    subgraph quorum [KRaft controller quorum]
        c1[Controller 1]
        c2[Controller 2]
        c3[Controller 3]
    end

    subgraph data [Broker data plane]
        b1[Broker 1 leader of some partitions]
        b2[Broker 2 leader of others]
        b3[Broker 3 leader of the rest]
    end

    player -->|bootstrap then produce to leader| b1
    landing -->|fetch from leader| b2
    tile -->|fetch ranges checkpointed by Spark| b3
    b1 --- c1
    b2 --- c2
    b3 --- c3
    c1 --- c2
    c2 --- c3
    c1 --- c3

Combined nodes draw the controller and the broker as one machine three times. Dedicated controllers draw the quorum off to the side, with brokers only on the data plane. Either diagram must show three metadata voters for a production bus, and more than one replica for each data partition.

mermaidCopy
flowchart LR
    subgraph partition [learning.progress partition 4]
        leader[Leader on broker 1 assigns offsets]
        f2[Follower on broker 2]
        f3[Follower on broker 3]
    end

    producer[Producer for learner_id hashing to 4] --> leader
    leader -->|replicate| f2
    leader -->|replicate| f3
    leader --> consumer[Landing consumer fetch]

If broker 1 stops and brokers 2 and 3 are in the ISR, the controller elects one of them. The producer refreshes metadata and appends to the new leader. Offsets do not restart. The landing consumer continues from the offset it committed. That sentence is the availability story you are buying with replication. A replication factor of one ends the story at "broker 1 stopped."

mermaidCopy
sequenceDiagram
    participant P as Producer
    participant B as Bootstrap broker
    participant L as Partition leader
    participant F as In-sync follower
    participant C as Controller quorum

    P->>B: metadata request using bootstrap.servers
    B-->>P: leader for learning.progress-4 is L
    P->>L: produce batch keyed by learner_id
    L->>F: replicate records
    F-->>L: caught up
    L-->>P: acknowledgement with base offset
    Note over C: leader identity lives in the metadata log
    Note over L: if L fails, C elects a new leader from the ISR

The acknowledgement line is only as strong as the settings in lesson 39. The diagram shows the path. It does not claim acks=0 waits for the follower.

Technical Explanation

Segment files and why retention is coarse

The leader appends to the active segment. When the segment reaches segment.bytes or segment.ms, the broker rolls a new active segment. Retention deletes old segments. It does not punch a single record out of the middle of a file, except through compaction, which rewrites segments to keep the latest key. For learning.progress the policy is deletion by time, because every observation matters. A record in a segment that is still open, or in a closed segment that is not yet old enough, remains readable. After a roll and a retention pass, a fetch for an expired offset fails.

That failure is the one lesson 32 named with failOnDataLoss. The architecture owes the Spark job a log that still contains the checkpoint's offsets. It owes the landing job the same for any uncommitted range. Fourteen days is a planning number we will set in lesson 39 only after we write down landing lag, the tile's recovery window, and a weekend. The mechanism to remember here is the segment: retention time is not a promise that a record vanishes at the millisecond it turns fourteen days old. It can live until its segment is eligible. Plan capacity for a full extra segment per partition, and plan replay for "the offset is gone," not for "the offset will disappear at a precise minute."

Indexes sit beside the segment. The offset index finds a record by offset. The time index finds an offset near a timestamp. A consumer that seeks to a timestamp uses that index and still has to know which timestamp type the topic stores. Seeking to broker time does not seek to event_time inside the JSON. If an operator uses a timestamp seek to repair landing, they must say which clock. The safe repair remains an offset from the manifest.

Leadership, epochs, and a log that does not go backwards

Each leader has an epoch, a generation number stored in the metadata. Fences based on that epoch stop a deposed leader from accepting a write after a new leader has been elected. You do not manage epochs in application code. You do notice them in logs when a producer retries and the old leader returns NOT_LEADER_OR_FOLLOWER. The client refreshes metadata and sends to the new leader. A hand-rolled HTTP post to "the Kafka server" has no such refresh. This is one reason the bus uses the official client rather than a one-off socket writer. The protocol, not your retry loop, knows how to abandon a stale leader.

High watermark is the largest offset that all members of the ISR have replicated. Consumers in the default isolation see records up to the high watermark, so they do not read a record that existed only on a leader that then died before replication. Transactional reads with read_committed add another fence, the last stable offset, which lesson 41 explains. For the plain progress path, the high watermark is the line that makes a fetch durable in the same sense the ISR is durable. A consumer cannot acknowledge business success for an offset the cluster has not made visible. The landing job's commit is a further step: the cluster has the record, and the object store has the bytes.

Request paths you will operate

A produce request names a topic partition, carries a batch, and returns either an error or the base offset of the append. The leader validates the epoch, appends, waits according to the acknowledgement mode, and responds. A fetch request names a topic partition and an offset, and the leader (or a follower, if you deliberately enable follower fetch) returns batches starting there. A metadata request returns brokers, topics, and leaders. An offset-commit request, which lesson 40 shows from the consumer, writes to __consumer_offsets. It does not move the high watermark of learning.progress. Mixing those two writes is the confusion lesson 32 spent a section untangling. The architecture keeps them on different topics on purpose.

List offsets tells you the earliest and latest offset of a partition. Lag, later, is latest minus the consumer's committed position, with care about what "committed" means. Earliest moving forward is retention doing its job. If earliest jumps because someone deleted and recreated the topic, every cursor is invalid. Recreating a topic is not a restart. It is a new log that happens to reuse a name. The landing manifest's offsets then point at different facts. Protect topic deletion with ACLs. A data engineer does not "clean up" learning.progress by deleting it on a Thursday.

Listeners and a bootstrap that matches the client

A broker can listen on more than one address: an internal listener for clients, an inter-broker listener, a controller listener. advertised.listeners is the address the broker tells clients to use for subsequent connections. A classic outage is a broker that bootstraps on a name the client can resolve and then advertises localhost or an RFC 1918 address the landing job's network cannot route. The symptom is a metadata fetch that succeeds and a produce that never connects. The fix is the advertised listener, not a second copy of the data.

Controller listeners are not client bootstrap addresses. Putting a controller port in bootstrap.servers yields a connection that does not speak the client protocol you expect. Platform documentation for this bus should hand data engineering three broker hostnames and the client port, and should keep the controller port on the platform diagram only.

Outside a closed lab, the client listener is not plaintext. SASL and TLS are SASL and SSL in the primer, and they are platform standards here. The progress principal gets WRITE on learning.progress and nothing on learning.enrollment. The landing principal gets READ on learning.progress and WRITE on learning.progress.dlq. The cohort principal, when it exists, gets READ on learning.enrollment. ALTER on the topic stays with the platform role that applies the reviewed configuration. Those are Apache Kafka ACLs, described in the official authorization documentation and practiced in /learn/kafka/acls. This lesson records the principal split so a shared "kafka-user" does not appear in the first manifest.

Capacity in one region

Planning assumption, not a benchmark: progress events are a few hundred bytes of JSON, with bursts while live lessons run, and enrollment is small beside them. Three brokers at replication factor three store about three times the retained bytes, plus indexes, with headroom for a replica rebuild. Disks sized to 95 percent full cannot accept that rebuild, and the ISR never heals. Capacity Planning is the worksheet. Measure this payload. Compression and TLS are worth turning on. Compression is a producer setting in lesson 38. TLS is a platform setting. Neither changes the log layout.

Ownership

Platform engineering runs the quorum, the disks, the listeners, the ACLs, and the upgrade. Data engineering owns the topic contract, the validator, the landing cursor, and the statement of how long offsets must remain readable. Both own the describe output in a review: if the replicas are not where the durability sentence requires, the topic does not go live because the application "already works on the lab broker." A single-node lab is how you learn the client. It is not how you accept the bus.

Practical Implementation or Design Walkthrough

Problem

Confirm that a cluster can host the progress bus: three metadata voters, brokers you can bootstrap, and a topic describe that shows a leader and followers for every partition of learning.progress. Then write the ownership note that goes in the service catalog. You are not choosing the final partition count in this lesson. You are proving you can see leaders and that a one-replica topic would be rejected.

What the platform process looks like

The following is Apache Kafka server configuration for a combined KRaft node in a three-node lab that mirrors the production shape. Hostnames and ports are examples. Security listeners replace PLAINTEXT before any real learner traffic. Do not treat this block as a Helm chart this course ships. It is the configuration surface the official broker understands.

propertiesCopy
# Combined KRaft node. Production client listeners use SASL_SSL, not PLAINTEXT.
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka1.internal:9093,2@kafka2.internal:9093,3@kafka3.internal:9093
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=PLAINTEXT://kafka1.internal:9092
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
log.dirs=/var/lib/kafka/data

Nodes 2 and 3 differ in node.id and advertised.listeners. controller.quorum.voters is the same list on every node. A client bootstrap list is kafka1.internal:9092,kafka2.internal:9092,kafka3.internal:9092. The controller port 9093 is absent from that list.

Describe the cluster with the official Admin client

Pin org.apache.kafka:kafka-clients to the patch your platform qualifies. The class names below are stable across the 3.7 through 4.0 clients. The example uses the 3.9 line because that client speaks to a KRaft cluster and matches the configuration keys in the 3.9 documentation. A Kafka 4.0 cluster still serves this client API for describe, produce, and consume.

javaCopy
import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.DescribeClusterResult;
import org.apache.kafka.clients.admin.TopicDescription;
import org.apache.kafka.common.Node;

import java.util.Properties;
import java.util.concurrent.ExecutionException;

public final class DescribeProgressBus {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
                "kafka1.internal:9092,kafka2.internal:9092,kafka3.internal:9092");
        // Timeouts fail the review loudly instead of hanging the ticket.
        props.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, "10000");

        try (Admin admin = Admin.create(props)) {
            DescribeClusterResult cluster = admin.describeCluster();
            System.out.println("clusterId=" + cluster.clusterId().get());
            for (Node node : cluster.nodes().get()) {
                System.out.println("broker id=" + node.id() + " " + node.host() + ":" + node.port());
            }
            TopicDescription progress = admin.describeTopics(java.util.List.of("learning.progress"))
                    .allTopicNames()
                    .get()
                    .get("learning.progress");
            progress.partitions().forEach(p -> System.out.println(
                    "partition=" + p.partition()
                            + " leader=" + p.leader().id()
                            + " replicas=" + p.replicas()
                            + " isr=" + p.isr()));
        }
    }
}

describeTopics returns leader, replica list, and ISR for each partition. A review fails the topic if any partition has an empty ISR, a leader of -1 (no leader), or a replica list of size one on a bus that must survive one broker loss. A lab topic created for the exercise can use one replica only if the write-up says "lab" in the first line.

The same facts are available from the distribution script:

bashCopy
kafka-topics.sh --bootstrap-server kafka1.internal:9092 \
  --describe --topic learning.progress

That script is Apache Kafka's. The output's leader and ISR columns are the ones you paste next to the contract. A GUI that hides the ISR is not a substitute in a durability review.

What you compare against the contract

For each partition, record leader id, replica ids, and ISR ids. Confirm replica ids are on distinct brokers, which the describe output shows as distinct node ids. Confirm bootstrap.servers uses the client listener. Confirm the principal that will produce cannot alter the topic. You still do not pick partition count here. If the topic does not exist yet, the describe call fails, and the failure is the signal to create it in lesson 39 with the replication settings, not to catch the exception and continue.

Expected output in a healthy three-broker lab

A six-partition topic with replication factor three prints six lines. Each line has one leader and three replicas. The ISR matches the replicas when the cluster is quiet. Leaders are spread across brokers rather than all sitting on broker 1. If every leader is on broker 1 after a restart, preferred leader election has not run; the data is still replicated, and the load is not. Platform runs preferred leader election. Data engineering notices the skew because the landing job's fetch latency may concentrate on one node. The primer's Leader Election page is the control. This walkthrough only requires you to see the concentration and to refuse to ignore it.

Operational note for the catalog

Write three sentences under the topic: the cluster id, the bootstrap list, and "deleting this topic invalidates every landing manifest and the Spark checkpoint." Name the on-call for broker disk and the on-call for landing lag. They are not the same rotation. An alert on under-replicated partitions pages platform. An alert on landing lag pages data engineering. Both alerts open the same incident if the cause is a dead leader and a consumer that cannot fetch. The describe output is the first artifact both rotations paste.

Enterprise Real-World Scenarios

Scenario 1: The single-broker demo becomes the live bus

A proof of concept ran kafka-server-start.sh on one virtual machine. The player integration worked. The tile moved. Someone promoted the same virtual machine because a rebuild would slip the course launch. The first host failure removes the only replica of every partition. Acknowledged progress events are gone. The landing manifests point at offsets the new empty volume does not have. The Spark checkpoint fails its source check, which is the behavior lesson 32 asked for, and it is also an outage.

The production shape is three combined KRaft nodes, or three brokers plus three controllers if the platform already runs them that way, with replication factor three on both bus topics. The demo VM stays a demo. Cutover is a new topic on the real cluster, a producer configuration change to the new bootstrap list, and a planned checkpoint start. Copying the VM's log.dirs onto one node of a three-node cluster does not create replicas. Replication factor is applied to partitions the cluster creates, not to files you copied by hand.

Security follows the same cutover. The demo used plaintext and an open listener. The production listener uses the platform SASL mechanism, and the player receives a principal that can write only learning.progress. A shared demo certificate pasted into production is how enrollment changes, once they exist, become writable by the player.

Cost is three machines and three disks instead of one. That cost is the price of the durability sentence lesson 36 already accepted. Saving it by running replication factor one on three brokers still loses a partition when its one replica's disk dies. The machines were not the expensive part. The replica count was.

Scenario 2: Metadata is healthy and the landing job still cannot write objects

Controllers have a quorum. describeCluster returns three brokers. Produce requests succeed from a bastion on the platform network. The landing job, in the data network, logs timeouts after metadata. advertised.listeners publishes broker addresses that only the platform network can route. The client learns the leader and then cannot open a socket to it.

The fix is an advertised listener on an address the landing job can resolve, or a second listener dedicated to that network, which is ordinary Kafka listener configuration. It is not a consumer timeout increase. Raising timeouts hides a black hole and lands records late, and the Monday handoff looks like lag. Platform changes the advertisement. Data engineering confirms with the same Admin describe from the landing network, then one produce of a synthetic event that the validator quarantines or the landing job writes to a scratch prefix. A successful describe from the bastion is not evidence the landing network works.

While this is broken, the tile in the platform network may still be fine. The consumer matrix predicted that. Do not reset the tile's checkpoint because landing cannot route. Two cursors, two networks, one log.

Scenario 3: A broker rebuild and a full disk

Broker 3's volume fails. The ISR for each partition shrinks to the remaining two replicas. With the durability settings lesson 39 will set, producers that wait for a minimum ISR of two still succeed. Landing continues. Platform replaces the volume and starts broker 3. The broker fetches from the leaders until it is in the ISR again. During the fetch, the two remaining disks hold the full retained log and the burst of live lessons.

If those disks were sized only for steady-state triple replication divided by three, they are already full, because each surviving broker holds its own replicas and the cluster cannot place the third replica. The rebuild stalls. Under-replicated partitions stay under-replicated. The next broker loss drops the ISR below the minimum and producers start failing. The architecture review asks for free disk equal to a rebuild, not for a green "disk used" chart at 92 percent on a quiet Tuesday. The primer's under-replicated partition page is the alert definition. The capacity sentence is the one this scenario adds to the catalog.

Best Practices

• Run a production metadata quorum of three voters. A single controller is a lab.
• Bootstrap clients with more than one broker address on the client listener. Leave controller ports off that list.
• Require a describe output that shows distinct replicas before you call the bus durable. Replication factor belongs to lesson 39; seeing the replica list belongs to this review.
• Keep unclean.leader.election disabled so a leader change does not rewind the log. Confirm the effective config, not the memory of a default.
• Size disks for retained bytes times replicas, plus a rebuild. A full volume is an availability incident waiting for the next failure.
• Separate principals for the player, the landing job, and topic administration.
• Treat topic deletion as a new log. ACLs should make deletion a platform action with a ticket, not a cleanup command.
• Expect NOT_LEADER_OR_FOLLOWER during elections. The official client refreshes metadata. A custom writer needs the same behavior or it does not belong on the path.
• Page under-replication to platform and consumer lag to data engineering, and teach both rotations to paste describe output.
• When the cluster is KRaft, retire ZooKeeper from the diagram. Do not run both because a slide still has the animal on it.

Common Mistakes and Anti-Patterns

One box labeled Kafka. The box hides quorum size, replica count, and listeners. Draw brokers, controllers, and one partition's leader and followers, or do not claim a failure story.

ZooKeeper on a Kafka 4 diagram. Kafka 4.0 removed it. Adding it creates a second system nobody is upgrading. Use KRaft. If the estate is still on an older ZooKeeper mode, say the version in the diagram and plan the migration with the primer, rather than copying a 4.0 mental model onto a 2.8 cluster.

bootstrap.servers set to a single load-balancer address that sends traffic to a random broker, with advertised listeners the client cannot reach. Metadata and data plane disagree. Fix advertisements and give the client more than one real broker.

Storing log.dirs for all brokers on one shared volume. Three processes, one failure domain, replication theater. Replicas belong on independent disks.

Using an internal topic as a scratch space. __consumer_offsets and __cluster_metadata are the cluster's own logs. A progress event written there is invisible to the contract and dangerous to the group protocol.

Recreating a topic to "empty" it. Offsets restart at zero. Manifests and checkpoints now name the wrong facts. If you must drop data, do it with a new topic name and a planned consumer cutover.

Reading followers by accident and expecting a feature. Default fetches go to the leader. Follower fetch is an explicit rack-aware configuration. Do not design the landing job around a behavior you did not turn on.

Treating the broker timestamp as the architecture's event time. The timestamp type is a topic property. The analytics clock remains the payload. An architecture that "uses Kafka time" has chosen LogAppendTime for business meaning and will mis-file catch-up traffic.

Troubleshooting and Design Review

Symptom: Admin.describeCluster fails from the landing job with a timeout. It succeeds from a bastion.

Likely cause: routing, security groups, or DNS for the bootstrap names. The cluster may be healthy.

Investigation: compare bootstrap hostnames from both networks. Check which port you used. A connection to 9093 is the controller listener and is the wrong test.

Corrective action: open the client listener path. Do not point the landing job at the controller port because it happened to accept a TCP connection.

Symptom: Metadata returns, produce fails with a timeout, and the leader host in the metadata is localhost or an address in a network the client cannot route.

Likely cause: advertised.listeners.

Investigation: print the node list from describeCluster and compare it with the address you successfully bootstrapped.

Corrective action: platform sets advertised.listeners to a client-routable name. Restart or reload according to the platform's broker procedure. Retry the produce from the landing network.

Symptom: describeTopics shows leader=-1 or an ISR smaller than the replica set for several partitions.

Likely cause: a broker is down, a disk is full, or a replica cannot catch up. Under-replication is the ongoing form of a small ISR.

Investigation: which broker id is missing from the ISR on many partitions at once. That id is the sick node. Check its process and its log.dirs volume.

Corrective action: platform restores the broker. Producers that require a minimum ISR may already be failing; that is the durability setting working. Do not enable unclean leader election to clear the alert. Do not reset consumer offsets. The log they committed is still the log.

Symptom: After a topic was deleted and recreated, the landing job writes objects whose offsets overlap last week's manifests, and the morning table mixes two days of facts under one offset range.

Likely cause: offsets restarted. The manifest key was reused for new bytes.

Investigation: cluster logs for topic deletion, and the earliest offset versus the manifest. A new topic has earliest offset 0 and a new topic id. TopicDescription includes a topic id on current clients. Compare it with the id you stored when the topic was created. A change means it is not the same log.

Corrective action: stop landing. Isolate the objects written after the recreation. Restore from a backup of the log only if platform actually has one. Otherwise cut over to a new topic name, accept the gap, and record it. Then remove delete permission from the data-engineering principal.

Symptom: All leaders sit on one broker after a rolling restart. Replication factor is three and the ISR is full. Fetch latency is uneven. Landing is still correct.

Likely cause: preferred leader election has not moved leadership back.

Investigation: the leader column in the describe output, compared with the first replica in each partition's replica list, which Kafka treats as the preferred leader.

Corrective action: platform runs preferred leader election. Data engineering confirms leaders spread and that lag did not jump. This is not a data-loss event. It is a load event. Do not rewrite partitions by hand.

A design review asks: How many controller voters are there, and what happens when one is gone? What is the bootstrap list, and which listener is it? For each bus topic, what are the replica sets, and are the disks independent? What is stored in the value bytes, and which clock is event_time? Who can delete the topic? Where does the landing job run, and can it route to the advertised leaders? Which page fires for under-replication, and which page fires for lag? A review that answers "we have Kafka" has not reviewed the architecture.

Interview Questions and Answers

Q1. What are the two planes in a current Kafka cluster, and which one stores learning.progress?

The data plane is the brokers, and they store the partition logs, including learning.progress. The metadata plane is the KRaft controller quorum, and it stores topic configuration, partition leadership, and the rest of the cluster metadata in a replicated metadata log. Producers append to the leader broker of a partition. They do not append learner events to the controller. Losing the metadata quorum stops elections and topic changes. Losing a single broker, when replicas remain in the ISR, does not have to stop the log.

Q2. Why is a topic not "an ordered queue"?

A topic is a set of partitions. Each partition is an ordered log with its own offsets. Records on different partitions have no order relative to each other. The progress contract gets per-learner order by hashing learner_id to a partition and keeping the partition count stable. Anyone who asks for a total order of all learners is asking for one partition, which throws away the parallelism the rest of the cluster is for. I would correct that on the diagram before agreeing to a consumer that sorts globally inside one process.

Q3. What does the leader do that a follower does not?

The leader assigns offsets and serves the produce path. Followers replicate the leader's log and sit in the ISR when they are caught up. By default they do not serve consumers. If the leader fails, a follower that is in the ISR can be elected, and clients refresh metadata. A follower that was out of sync is the wrong election target for this bus, because it may be missing acknowledged writes. That is why unclean leader election stays off.

Q4. What is bootstrap.servers, and why is a controller port a mistake there?

It is the list of client-listener addresses the producer or consumer uses to fetch initial metadata. After that, the client talks to partition leaders at the addresses those leaders advertise. The controller listener speaks the quorum protocol, not the client produce/fetch protocol you want. I give the client two or three broker addresses so one down broker does not make the bootstrap itself a single point of failure. I still need advertised listeners the client can route to, or the metadata will name a leader the client cannot reach.

Q5. How does KRaft change the picture you may have learned with ZooKeeper?

The controller quorum is Kafka processes replicating a metadata log. There is no separate ZooKeeper ensemble on a Kafka 4.0 cluster. Brokers learn leadership and topic metadata from that quorum. Clients still use bootstrap.servers against brokers. I would not draw ZooKeeper on a new design. If an existing cluster still uses it, I would name the version and treat migration as platform work, using the primer's comparison, rather than mixing the two mental models in one runbook.

Q6. What is the high watermark, in pipeline language?

It is the highest offset that the ISR has fully replicated. Consumers do not read past it, so a record that lived only on a leader that died before replication is not handed to the landing job as a committed log fact. The landing job then has its own commit, which records that those bytes are in the object store. The high watermark is a cluster property of the partition. The group offset is a consumer property. They are different numbers, stored for different reasons.

Q7. Why can retention delete an offset the Spark checkpoint still names?

The broker deletes old segments according to retention. The checkpoint is a directory Spark owns, not a pin on the broker. If the checkpoint's offset falls below the earliest offset still on the partition, the source cannot reconstruct the input. The architecture has to keep retention longer than the tile's downtime and the landing lag. failOnDataLoss is how Spark refuses to skip that gap. Recreating the topic causes the same class of failure immediately, because the old offsets are simply gone.

Q8. What do you look for in describeTopics before you call the bus ready?

A leader that is a real broker id, a replica list of size three on independent brokers, and an ISR that matches that list while the cluster is healthy. I also check that leaders are not all piled on one broker after a restart. I do not treat a successful produce from a lab client on one node as the same evidence. The describe output is the evidence that matches the durability sentence.

Q9. Who owns a broker disk filling up, and who owns a landing manifest falling behind?

Platform owns the disk, the replica placement, and the under-replicated alert. Data engineering owns the landing cursor and the lag of that group. A full disk becomes everyone's incident when it shrinks the ISR and produce starts failing. The ownership split still matters, because the first actions differ: free or replace a volume, versus rewind or hold a consumer. I do not want one generic "Kafka on-call" that restarts brokers and resets offsets in the same breath.

Q10. A stakeholder asks to query the last year of progress in Kafka SQL. What do you offer instead?

The log retains a short window so consumers can fan out and replay. The lakehouse raw zone is the long copy, after landing commits. I offer the raw objects and course_day, which already have a year-scale retention and a query engine. I do not extend topic retention to a year to avoid building the landing job. That would triple a year of bytes on the brokers and still would not be the SQL warehouse. The cluster is the bus. The warehouse is the place you query history.

Hands-On Exercises

Exercise: Draw the bus cluster and reject a one-replica describe

Problem statement. You are given a three-node KRaft lab, or a written describe output if you cannot run brokers. Prove you can identify leaders, replicas, and a bootstrap mistake. Write the catalog note for learning.progress.

Tasks.

1. Draw the data plane and the controller quorum for a three-node combined cluster. Mark a client listener and a controller listener as different ports.
2. Using Admin.describeTopics or kafka-topics.sh --describe, capture leader, replicas, and ISR for every partition of a lab topic. If you cannot run a cluster, annotate the sample output your instructor provides and state what you would have called.
3. Identify one condition that fails the durability review: a replica list of size one, an empty ISR, or leader=-1. Write the sentence you would put on the ticket.
4. Explain a bootstrap list that includes only port 9093, and what you replace it with.
5. State what happens to offsets if the topic is deleted and recreated, and how a landing manifest would mislead the morning job.
6. Name the two on-calls and the first artifact each pastes.
7. In one paragraph, separate this diagram from the consumer matrix in lesson 36. The diagram does not choose group ids. The matrix does not choose node.id.

Expected deliverables. The diagram, the describe transcript or annotated sample, the rejection sentence, and the catalog paragraph.

Evaluation rubric.

You pass when a reviewer can point to the ISR on your transcript and say whether one broker loss would still leave an in-sync copy.

Lesson Summary

A Kafka cluster for this bus is brokers storing partition logs and a KRaft quorum storing metadata. Kafka 4.0 does not add ZooKeeper to that picture. A topic is many ordered partitions. Offsets are positions inside one partition. Producers and consumers talk to the leader of a partition. Followers replicate, form the ISR, and become eligible leaders. The high watermark is what makes a fetch match that replication. bootstrap.servers is a client-listener bootstrap, not a controller address, and advertised listeners must be routable from the landing job. Internal topics are not scratch space. Deleting learning.progress mints a new log under an old name and invalidates manifests and checkpoints. Three voters and three independent replicas are the production shape that matches lesson 36's promise to survive one broker loss. Partition counts, retention, and the acknowledgement settings that wait for the ISR are next.

Kafka Producers and Consumers writes and reads learning.progress with KafkaProducer and KafkaConsumer. Replication numbers and retention are Kafka Topics, Partitions and Replication. The operator primer remains Kafka Architecture.

Further Reading

• Apache Kafka documentation, design — partitions, replication, and the log as the project specifies them.
• KRaft — controller quorum, combined versus dedicated roles, and the metadata log. Read this before you copy a ZooKeeper runbook onto a new cluster.
• AdminClient Javadoc — describeCluster and describeTopics, the API the walkthrough uses. Match the Javadoc to the kafka-clients patch you pin.
• ZooKeeper vs KRaft, Brokers, and Replication — the specialty primer for operational depth this lesson deliberately leaves out.
• Streaming Data Architecture — the contract the cluster is hosting.