Kafka Topics, Partitions and Replication

Data Engineering & Modern Data Platforms · Lesson 39 · Real-Time Data Engineering and Kafka

Lesson Overview

The player from Kafka Producers and Consumers can append learning.progress with acks=all. That acknowledgement is only as strong as the topic underneath it. A lab topic created with one partition and one replica will accept the same client and will lose the acknowledged hour when its one disk dies. A topic created with two hundred partitions "so we never resize" will hash learner_id across a map you cannot shrink, will make every deploy a large rebalance, and will still put a hot key on one partition if someone later keys by course_id. A topic that keeps progress for seven days because that is the broker default will delete the offsets a Spark checkpoint still names after a long weekend, which is the failOnDataLoss failure Structured Streaming with Spark already told you to expect.

This lesson sets partition count, key behavior, replication, minimum in-sync replicas, and retention for the two bus topics. It also decides cleanup policy: delete for progress, because every observation matters, and compact for enrollment only if a consumer truly wants the latest status from the log itself. The morning table does not. Ordering stops at the partition boundary. Adding partitions does not reshuffle old data and does change the hash of new keys. Those are the decisions. Broker internals of the replica fetcher sit in Replication, ISR, Retention Policies, and Log Compaction. Read them when you are tuning a cluster. Write the topic ticket from this lesson when you are shipping the bus.

Learning Objectives

By the end of this lesson you will be able to:

1. Choose a partition count from consumer parallelism and operational cost, and keep the progress key as learner_id.
2. State the ordering boundary: per partition, per key only while the partition count is stable.
3. Set replication factor 3, min.insync.replicas 2, and rely on the producer's acks=all as one durability policy.
4. Explain why unclean leader election stays off for this bus.
5. Size retention to landing lag, checkpoint recovery, and a named replay window, and describe why deletion is segment-granular.
6. Choose delete cleanup for progress and justify compaction only for a current-state enrollment log.
7. Describe what adding partitions does to existing offsets and to the default partitioner's mapping.

Prerequisites

• Apache Kafka Architecture: leaders, followers, ISR, segments, and the high watermark.
• Kafka Producers and Consumers: the key, acks=all, and idempotence already set on the client.
• Streaming Data Architecture: who must be able to replay, and for how long you promised to think about it.
• org.apache.kafka.clients.admin.Admin and org.apache.kafka.clients.admin.NewTopic, plus org.apache.kafka.common.config.TopicConfig. The shell equivalent is kafka-topics.sh from the Apache Kafka distribution.
• /learn/kafka/partitions and /learn/kafka/partition-strategy for the primer's longer catalog of partitioners. This lesson uses the default partitioner and refuses a custom one unless the hash is written down and tested.

Detailed Core Concepts

A partition is the unit of order, parallelism, and replication

Each partition is one log, replicated as a unit, consumed by one member of a group at a time. More partitions mean more parallel consumers, more replica directories, more open files, and a larger rebalance when a member leaves. They do not mean more order. They mean the opposite: more boundaries across which order does not exist.

For learning.progress the parallelism that matters is the landing job and any future group that must keep up with a live lesson burst. The tile's parallelism is Spark's read of the same partitions, inside a micro-batch, and it does not use a group. A planning assumption, labeled as such: a handful of landing instances can drain the burst if each instance owns several partitions, and twelve partitions is a starting count you can operate. Twelve is not a law. It is a number you can spread across three brokers (four leaders each after a preferred election), assign to up to twelve landing threads if you ever need them, and still explain on a diagram. Sixty partitions on this bus would be a hypothesis you measure into, not a default you copy from a larger company. Two hundred would be a cost: rebalances, controller metadata, and a hash space you will regret if a key scheme changes.

learning.enrollment is smaller. Six partitions, or even three, keep a key's changes ordered and give the cohort job room to run more than one instance. Matching the progress topic's width is unnecessary. Sharing its partition count so "all topics look the same" couples two capacity plans.

The key stays learner_id for progress. The analytics grain course_id is a shuffle inside Spark, not a partition key. A popular course with tens of thousands of simultaneous learners would otherwise be one partition's problem, and the learners in that course would be totally ordered with each other for no product reason. Hot-key pain that remains after a correct key is a single learner's client gone wild, which is a quota on the player, or a bug that emits in a loop. It is not a reason to partition by course.

The default partitioner and a stable modulus

DefaultPartitioner hashes the serialized key with Kafka's murmur2 and takes a non-negative value modulo the number of partitions. The same bytes map to the same partition while that number is constant. Null keys use the sticky partitioner and are rejected by our player. A custom partitioner is a second hash you must now keep forever, including in every replay tool. Do not write one for this bus. If you need a different spread, change the key you hash, document it as a contract change, and start a new topic so old and new mappings do not share a name.

Adding partitions keeps every existing record where it is. New records are hashed against the new count, so a learner who lived on partition 4 may continue on partition 4 or may land on partition 15 for the next event. Their history is now split. Per-learner order across the split is gone. Consumers that need that order must read both partitions and merge by event_time and event_id, which is a new product, or you must drain, freeze, and cut over to a new topic with the new count and a dual-write window. The cheap moment to pick the count is before the player ships. The expensive moment is a Thursday during a live course.

You cannot reduce the partition count of an existing topic in place. Planning "we will lower it if twelve is too many" is planning a new topic. Start at the count you can run, not at the count you might need in five years.

Replication factor, minimum ISR, and acknowledgements

Replication factor is how many replicas of each partition exist. Three independent brokers, factor three, one replica on each broker: loss of one broker leaves two copies. min.insync.replicas is how many of those replicas, including the leader, must acknowledge a produce when the producer uses acks=all. Set it to 2. Then:

• One broker down: the ISR has two members, produces succeed, landing continues, platform rebuilds the third replica.
• Two brokers down: the ISR has one member, which is below the minimum, and acks=all produces fail with NotEnoughReplicasException or a timeout. The log stops accepting new progress rather than acknowledging a write that has nowhere to replicate.

That failure is the policy. The alternative, min.insync.replicas=1 with acks=all, acknowledges as soon as the leader has the record even if it is the only replica left. You have then paid for three copies in steady state and given them up exactly when the cluster is already wounded. The player surfaces a retryable error. The lesson client keeps the event. That is a better user story than a green acknowledgement and a missing offset.

acks=all on the producer is required for the minimum ISR to matter. A misconfigured player with acks=1 will not wait, no matter what the topic says. The topic config and the producer config are one policy stored in two places. Review both.

unclean.leader.election.enable stays false. If the ISR is empty because every in-sync replica is down, the partition has no leader rather than a stale one. The partition is unavailable. It does not rewind. A rewind would make landing manifests point at offsets whose bytes changed identity, which is worse than downtime. Availability of the live tile does not outrank the identity of an offset. The tile waits. The batch waits. Support says the bus is down.

Retention for a log that other systems point at

retention.ms on learning.progress must cover:

• landing lag you are willing to alert on and still recover, not the lag you hope for
• the longest Spark checkpoint outage you intend to resume without a new checkpoint and a deliberate gap
• the watermark delay from lesson 32, which is minutes and is the small term
• a weekend plus a deploy, if the only person who can fix landing is human

A planning value of fourteen days (1209600000 milliseconds) covers a week of failure and a week of argument. It is a choice, written on the topic, not the broker default of seven days applied by accident. If landing is down for fifteen days, the offsets are gone and the honest incident is a gap. Lengthening retention after the segments are deleted does not bring them back.

Retention deletes closed segments, not individual records on their birthday. segment.bytes and segment.ms decide how large and how old a segment is before it rolls. A very large segment rolls rarely, so a record can outlive retention.ms by almost a segment. A very small segment rolls often, creates many files, and makes deletion closer to the retention clock. The broker default segment is large (1 GiB) and also rolls on time (segment.ms, defaulting from log.roll.ms, commonly seven days). For a moderate progress topic, the time roll matters more than the size roll. Leave the size default unless you have measured file-handle pressure. Set segment.ms to a day if you want deletion to track a fourteen-day policy within about a day rather than within a week. Document the slop. Do not promise a lawyer that a record disappears at a precise millisecond. Privacy deletion of a specific learner is not a retention setting. It is a rewrite or a crypto-shredding design this lesson does not pretend the log provides. If you must delete one learner's events inside the window, you are building a compaction or a filter in the lake, and you still cannot purge a replica that has not rolled.

retention.bytes caps a partition. A bytes cap that hits before the time cap will delete offsets the checkpoint needs while the topic still "has fourteen days" in the config. Do not set a bytes cap on learning.progress unless you have an alert at a threshold that leaves the replay window intact. Time is the policy. Disk headroom from lesson 37 is how you afford it.

Delete versus compact

cleanup.policy=delete on progress. Every event is a fact. Compaction would keep only the latest record per learner_id and throw away the path from 10 percent to 40 percent to 100 percent. The tile and course_day would under-count and you would not have a DLQ to explain it. Compaction is the wrong cleanliness.

learning.enrollment may use cleanup.policy=compact if the cohort job and a rebuild tool want the log to converge to the latest record per key. The key must be the enrollment identity, learner_id plus course_id, which lesson 36 already chose. Compaction does not preserve every change. It preserves the latest. A consumer that needs the full change history for audit uses a delete-retention topic, or the lake, not the compacted log. Tombstones are null values. The cohort op=delete is a payload, not a Kafka tombstone, unless you also publish a null value. If you compact, publish a null value for a delete after the payload record, or accept that compaction will keep the delete payload forever as the latest value, which is actually what you want: the latest value says op=delete. A null tombstone would remove the key entirely after delete.retention.ms, and a rebuild would not see the delete if it started from a blank table and only read the compacted log. For a rebuild from zero, the latest record should be the delete payload, not an absent key. So if you compact enrollment, do not tombstone with a null value. Let the delete record be the latest value. A consumer folding from the beginning of the compacted log still needs a snapshot if compaction has dropped intermediate states and the consumer's logic is not "latest wins." Latest wins is exactly the cohort fold if LSN is compared. Intermediate LSNs are gone. That is acceptable for current state. It is not acceptable for an audit of every status the learner passed through. Audit reads the lake landing of enrollment, with delete cleanup and a longer life in object storage.

Two topics, two policies. Do not set compact,delete on progress to be clever. The combination deletes old keys and compacts the rest, and it will drop observations.

What replication does during a produce

The leader appends the batch to its log and waits, under acks=all, until the number of in-sync replicas is at least min.insync.replicas and those replicas have fetched the record. Followers pull. The leader does not push a synchronous replica commit to every broker in the cluster, only to the ISR policy you set. A follower outside the ISR is not a copy you just promised. It is a copy trying to catch up. Lagging followers are an under-replicated alert when they fall out, or a silent catch-up when they are still inside the time window. replica.lag.time.max.ms (broker config, default 30 seconds) is the clock that drops them. You do not set a message-count lag threshold. That older control is gone. A slow disk that still fetches within the time window stays in the ISR and slows acks=all. That shows up as produce latency, not as data loss. It is still an incident if the tile's producers start missing their callback budget.

Preferred leader election moves leadership back to the first replica in the assignment list after a restarted broker catches up. Without it, leaders pile up on the brokers that stayed up. The data is safe. The load is not. Platform schedules the election. Data engineering checks the describe output from lesson 37 after a rolling restart.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph progressTopic [learning.progress RF 3 cleanup delete retention 14d]
        p0[Partition 0 leader and two followers]
        p1[Partition 1]
        pN[Partitions 2 through 11]
    end

    key[learner_id murmur2 mod 12] --> p0
    key --> p1
    key --> pN

    subgraph enrollTopic [learning.enrollment RF 3 compact or delete]
        e0[Partition 0 latest record per enrollment key]
    end

    enrollKey[learner_id plus course_id] --> e0

    p0 --> landing[Landing group one member per owned partition]
    p0 --> tile[Spark source checkpoint not a group]

Twelve progress partitions are the planning start. The diagram's job is to show that the modulus and the replica factor are explicit, and that the tile does not take a seat in the landing group.

mermaidCopy
sequenceDiagram
    participant Producer
    participant Leader
    participant ISR as Second ISR replica
    participant Out as Replica outside ISR

    Producer->>Leader: produce acks=all
    Leader->>ISR: follower fetch catches up
    Note over Leader: in-sync count is 2 which meets min.insync.replicas
    Leader-->>Producer: success and offset
    Leader--xOut: this replica is not part of the ack
    Note over Out: catch-up later or stay under-replicated

If the second ISR replica is also down, the leader refuses the produce. The producer callback fails. The player keeps the event.

mermaidCopy
flowchart LR
    subgraph time [Progress retention]
        active[Active segment not deleted]
        closed[Closed segment older than retention.ms]
        gone[Deleted segment offsets fetch as out of range]
    end
    active --> closed --> gone
    checkpoint[Spark checkpoint offset] --> gone
    gone --> fail[failOnDataLoss stops the tile]

That last arrow is a successful safety property. Silencing it by skipping missing offsets writes a tile that looks continuous and is not.

Technical Explanation

Topic configuration we will set

The DLQ is smaller because invalid bursts should be rare. It is not replication factor one. A poison payload you cannot afford to lose during a broker failure is the evidence for the incident. Enrollment stays on delete until someone writes the rebuild consumer that can start from latest-per-key and proves it against a snapshot. Turning on compaction the same day you create the topic, without that consumer, saves disk and surprises the first audit.

message.timestamp.type stays CreateTime so the broker records the producer's timestamp. Analytics still use payload event_time. The broker timestamp remains the operational column in the raw line.

Creating the topic with Admin

javaCopy
import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.common.config.TopicConfig;

import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutionException;

public final class CreateProgressTopic {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, args[0]);
        short replication = 3;
        NewTopic progress = new NewTopic("learning.progress", 12, replication);
        progress.configs(Map.of(
                TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2",
                TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_DELETE,
                TopicConfig.RETENTION_MS_CONFIG, Long.toString(14L * 24 * 60 * 60 * 1000),
                TopicConfig.SEGMENT_MS_CONFIG, Long.toString(24L * 60 * 60 * 1000),
                TopicConfig.UNCLEAN_LEADER_ELECTION_ENABLE_CONFIG, "false"
        ));
        try (Admin admin = Admin.create(props)) {
            admin.createTopics(java.util.List.of(progress)).all().get();
        }
    }
}

NewTopic is the official admin type. The same map is what kafka-topics.sh --create sets with --config. Run it as the platform principal. The player principal must not be able to --alter these configs. A later "temporary" drop of min.insync.replicas during an incident is a durability change and belongs in the incident channel, with a revert, not in a laptop history nobody reviews.

kafka-configs.sh --describe --topic learning.progress --all shows effective values, including broker defaults you did not set. Read the effective min.insync.replicas and unclean.leader.election.enable. A topic created before this lesson can look fine in a diagram and still carry the broker default of min.insync.replicas=1.

Changing partitions later

admin.createPartitions increases the count. It does not move existing records. After the increase, publish a known learner_id in a lab and observe which partition the new record uses. Compare with the partition of that learner's previous offset. If they differ, you have demonstrated the split. Do this in a lab before you ever do it in production, and prefer a new topic for a planned reshuffle so the landing prefix can record a topic id change as a new series rather than a silent modulus change.

Hot partitions

Twelve partitions spread learners only as well as the hash spreads them. One course's learners are many keys, so they spread. One key that emits thousands of events a second is one partition at 100 percent and eleven idle. The landing member that owns that partition falls behind. Lag on one partition is the symptom. The fix is to stop the hot key at the player, not to add partitions, because the hot key still hashes to one of the new partitions. Adding partitions helps when many keys are collectively too fast for the current consumer count. It does not split a key. If you truly must split a key, you are changing the ordering contract, and you should say so.

Skew is visible in bytes-in per topic-partition metrics. A review that only watches topic-level bytes-in will miss it until the tile's watermark stalls on one input task. Spark will show the same skew if one partition is huge inside a trigger. maxOffsetsPerTrigger limits the trigger. It does not rebalance Kafka partitions.

Replica placement

Kafka's rack-aware assignment can place replicas in different racks if broker.rack is set. On a single-zone bus, different brokers and different volumes are the requirement from lesson 37. On a design that claims two availability zones, a replication factor of three with two replicas in one zone still loses quorum of copies when that zone dies, depending on placement. Do not claim zone survival without looking at the replica list. The describe output is the check. A spreadsheet of intentions is not.

The Spark and landing retention contract

Write this next to retention.ms: "Offsets required by the landing group and by the activity-tile checkpoint must remain on the broker. failOnDataLoss stays at its default fail behavior. If earliest offset passes a checkpoint, stop and choose a gap explicitly." Fourteen days is the number. The sentence is the control. Shortening retention is a schema-level change for those cursors, even though it is only a config value.

Practical Implementation or Design Walkthrough

Problem

Create learning.progress and the DLQ with the table above. Prove that a produce with acks=all fails when the ISR is too small, in a lab where you can stop brokers. Prove that two events for the same learner_id share a partition. Write the retention sentence into the catalog.

Steps

1. Create the topics with CreateProgressTopic or kafka-topics.sh --create and the same configs. Describe them and paste leader, replicas, ISR, and the effective configs.
2. Produce two records for lrn_1842 with the lesson 38 producer. Confirm both metadata partitions are equal.
3. Produce two records for lrn_1842 and lrn_9999. Confirm the partitions are allowed to differ. Order across learners was never the contract.
4. In a three-broker lab, stop one broker. Produce again with acks=all. The produce should succeed, and describe should show an ISR of two for the affected partitions.
5. Stop a second broker. Produce again. The callback or get should fail. Restart the brokers and confirm the ISR returns to three without unclean election and without the offset going backwards. The last successful offset is still the last record you can fetch.
6. Document retention.ms, segment.ms, and the consumers that depend on them. State that compaction is not enabled on progress, and why a latest-per-learner compact would destroy the tile.

Expected evidence

The describe transcript, the two partition numbers for one learner, the exception class from the double-broker stop (NotEnoughReplicasException, TimeoutException, or a retriable wrapper around them, depending on timing), and the catalog sentence. A test that only creates the topic and never stops a broker has not exercised the policy. If you cannot stop brokers, say so, and still compute from the config what the second failure must do. Do not flip min.insync.replicas to 1 to make the lab green.

Enrollment topic, created and left on delete

Create learning.enrollment with six partitions, replication three, minimum ISR two, delete cleanup, fourteen-day retention. Do not enable compaction in this walkthrough. Write a note: "Compaction is a later decision if the latest delete payload must survive as the only copy in Kafka. The lake landing of changes is the audit copy." That note prevents a well-meaning operator from compacting the topic over a weekend to save disk.

Enterprise Real-World Scenarios

Scenario 1: Fourteen days, a holiday, and a checkpoint

The tile's driver is down from 23 December through 2 January. Landing is also down, because the same credentials expired and the person who rotates them is away. retention.ms is seven days, the broker default nobody overrode. On 2 January the brokers have deleted the segments. The checkpoint names offsets that earliest has passed. Spark fails the query. course_day has a hole for the holiday week because landing never wrote the objects, and the log cannot fill it.

Fourteen days would have covered this particular holiday. Twenty days would cover a longer one. The point of the scenario is the coupling: retention is a product decision about how long a human outage may be, not a broker default. The cost is disk, about twice the seven-day footprint, times replication three. Buy the disk or staff the holiday. Do not discover the choice in January. The security angle is the expired credential: a principal that cannot read is the same outcome as a full disk from the checkpoint's point of view. Rotate credentials on a calendar that is shorter than retention, and alert on lag while the offsets still exist.

Scenario 2: A partition increase during a live course

Traffic grows and someone runs kafka-topics.sh --alter --partitions 24 on learning.progress at 10:00 UTC, in the middle of a lesson. Old offsets stay. New events for half the active learners hash to the new partitions. The landing job still writes correct per-offset objects. The per-learner sequence is split. A downstream "latest progress per learner" built by taking the max offset per key, which was already a shaky idea across partitions, now misses the continuation on the new partition if the job only tails partitions it saw at startup. The tile, which groups by course_id and event time, is largely fine because it does not assume one partition per learner. The cohort-style progress debugger, which assumed one partition per learner, is wrong.

The repair is not to alter the count back. You cannot. The repair is to treat 10:00 UTC as a contract break: readers that need per-learner order merge by event_time for the overlapping generation, or you cut over to a new topic at a quiet hour and dual-write until consumers move. The scenario's lesson is to make partition changes a reviewed contract event. The ACL that forbids the player from altering topics should also forbid a shared human principal from altering them without the platform role.

Scenario 3: Compaction "to keep enrollment small" drops the audit

An operator sets cleanup.policy=compact on a delete-retention enrollment topic that the audit job reads from the beginning each night to build a change history. After the cleaner runs, intermediate statuses are gone. The audit shows the final status and invents a story that the learner never passed through waitlisted. The lake copy, if it was landing every change, still has the path. The log does not.

If the lake copy also was not built, the path is gone inside the retention interaction of compaction. The fix going forward is delete cleanup plus lake landing for audit, and compaction only for a consumer whose definition is latest-wins and that has been tested against tombstones or delete payloads. The fix for the missing week is the lake, or an honest gap. Replaying a compacted log will not restore intermediate keys.

Best Practices

• Fix the partition count before producers go live. Treat an increase as a new mapping, not a capacity tweak.
• Key by the order you need. Keep course_id in the payload for the tile's shuffle.
• Set replication factor 3, min.insync.replicas 2, producer acks=all, unclean leader election false. Review them as one policy.
• Set retention.ms from the slowest cursor you will resume, and set segment.ms so the slop is about a day, not a week.
• Leave retention.bytes unset on progress unless an alert protects the replay window.
• Use delete cleanup for observations. Compact only a current-state log whose consumers have been written for latest-per-key.
• Spread leaders with preferred leader election after restarts. Check describe output, not a hope.
• Alert on under-replicated partitions and on per-partition lag. Topic-level averages hide a hot partition.
• Forbid topic alter and topic delete for application principals.
• Store the topic id from describeTopics in the catalog when you create it. A recreation changes the id and restarts offsets.

Common Mistakes and Anti-Patterns

One partition so the topic is "totally ordered." You serialize every learner behind one leader and one consumer. The order you needed was per learner. The order you bought is a bottleneck. Use twelve partitions and a key.

Two hundred partitions on day one. Rebalances and file handles grow. You cannot shrink. You do not yet have the consumers to read two hundred ways. Start at a count you can draw.

Partition key equal to the dashboard group key. course_id melts a popular course onto one partition. Hash learner_id. Let Spark group by course.

acks=all with min.insync.replicas=1. The acknowledgement waits for a minimum of one, which is the leader. The replicas are decoration until you raise the minimum. Set 2.

Unclean leader election to clear an alert. The log can rewind relative to acknowledged offsets. Manifests and checkpoints describe bytes that are no longer there. Take the outage. Restore replicas.

Seven-day default retention. It is a default, not a decision. The checkpoint and the holiday scenario will find it. Write fourteen days, or a different number you calculated, onto the topic.

Compacting the progress topic. You keep the latest event per learner and delete the course. The tile cannot be rebuilt. Delete cleanup is the policy.

Adding partitions during a live lesson. New events split per-learner order. Existing data stays. There is no in-place undo. Schedule a new topic if the mapping must change.

A bytes retention cap nobody monitors. It deletes the replay window while the time setting still says fourteen days. Time is the policy. Disk is the capacity plan.

Troubleshooting and Design Review

Symptom: Produce latency jumps above the player's budget. ISR is full. One broker's disk is busy. acks=all is on.

Likely cause: a slow in-sync replica still inside replica.lag.time.max.ms. The leader waits for it because it is still in the ISR.

Investigation: per-broker disk and fetch-follower metrics. Produce time on the client. Describe still shows a full ISR, so this is not the under-replicated page.

Corrective action: relieve the disk. Do not drop acks to 1 to hide it. If the replica falls out of the ISR, latency may recover and under-replication will page. That transition is the cluster protecting the acknowledgement. Replace the volume if the disk is the fault.

Symptom: NotEnoughReplicasException during a broker replacement. One broker is down on purpose. The topic's minimum ISR is 2 and replication factor is 3. ISR size is 2. Produces should work.

Likely cause: a second replica is also out of sync, or the topic you are producing to is not the one you altered. A lab topic with replication factor 2 and minimum 2 cannot lose any broker.

Investigation: describe the exact topic. Read ISR size per partition. Read the effective min.insync.replicas.

Corrective action: restore the unexpected second failure. If the topic was created with the wrong factor, create the right topic and cut over. Do not lower the minimum to keep the demo moving and then leave it.

Symptom: The tile query fails. The exception says offsets are out of range. Landing objects stop at an older offset. retention.ms is 604800000.

Likely cause: seven-day retention passed the checkpoint. Segments are gone.

Investigation: earliest offset per partition versus the checkpoint. The catalog's retention sentence, if it exists, was not applied.

Corrective action: do not set failOnDataLoss to false. Choose a new checkpoint start at the earliest available offset or at a known landing boundary, and record the gap in course_day. Then set retention to the reviewed value. The missing days stay missing unless the lake already has them.

Symptom: After --alter --partitions, a learner's events alternate partitions. Support says progress "went backwards" in a debugger that takes the max offset per partition and ignores the other partitions.

Likely cause: the modulus changed. The debugger assumed one partition.

Investigation: metadata partition for a new event versus the partition of the previous stored offset for that learner_id.

Corrective action: fix the debugger to merge on event_time. Plan a quiet cutover if you need a single partition per learner again. Do not delete the topic.

Symptom: Enrollment audit is missing intermediate statuses. cleanup.policy is compact.

Likely cause: the cleaner dropped old keys.

Investigation: topic config and a consumer reading from the earliest offset. Count records per enrollment key. A compacted current-state log has one.

Corrective action: stop using that topic as the audit. Land changes under delete cleanup to the lake. If you need current state in Kafka, keep a separate compacted topic fed by a job that writes the latest payload, including deletes, and document that it is not history.

A design review asks: What is the partition count, and what consumer count does it allow? What is the key, and what order does it not give? What happens to a learner's partition if we add partitions? What is replication factor, minimum ISR, acks, and unclean election, as one policy? What is retention.ms, what is the segment slop, and which cursors must still resolve? Is compaction on, and which consumer definition requires it? Where are the replicas physically? A review that says "standard topic" has not answered.

Interview Questions and Answers

Q1. What does a partition buy you, and what does it cost?

It buys an independent ordered log, one consumer-group member of parallelism, and a replication unit. It costs file handles, rebalance work, and another boundary across which records are unordered. For progress I start at twelve on a three-broker bus so landing can scale out and leaders can spread, and I refuse a count I cannot draw. I do not add partitions to create a global order. Global order would be one partition, and I do not want that either.

Q2. Why is the key learner_id and not course_id?

Per-learner order is the contract. course_id is the tile's aggregation grain. Hashing the grain puts an entire popular course on one leader and one landing thread. The tile can shuffle by course_id after the read. The log should spread learners. A hot learner_id is a client bug or a quota problem, and adding partitions will not split that key.

Q3. What happens to existing data and to new keys when you increase the partition count?

Existing records stay on their partitions with their offsets. The default partitioner hashes new keys modulo the new count, so a learner's next event may land on a different partition. Per-learner order is split from that moment. There is no in-place decrease. I treat the change as a contract event, and for a planned reshuffle I prefer a new topic and a cutover to a silent --alter during a lesson.

Q4. Explain replication factor 3 and min.insync.replicas 2 with acks=all.

Three replicas sit on three brokers. A successful acknowledgement means at least two in-sync replicas have the record. One broker can be down and produces still succeed. Two brokers down and produces fail rather than acknowledge a single copy. acks=all is what makes the producer wait for that minimum. acks=1 would ignore it. Unclean leader election stays off so we do not elect a replica that never had those acknowledged writes.

Q5. Why is a slow replica still in the ISR a latency problem and not a durability win you should disable?

The ISR time window keeps a replica in the set while it is fetching within replica.lag.time.max.ms. acks=all waits for it. That protects the acknowledgement. Removing the wait because latency hurts would acknowledge on the leader alone. I would rather see produce latency and fix the disk. If the replica falls out, the set shrinks, latency may drop, and under-replication pages. Both states are operable. Silent acks=1 is not.

Q6. How do you pick retention.ms for learning.progress?

It must exceed the landing outage and the Spark checkpoint outage I am willing to resume, plus segment roll slop. I use fourteen days as a planning value so a holiday week plus a slow repair still leaves the offsets in place, and I set segment.ms to a day so deletion tracks that policy within about a day. Seven days is the broker default, not a calculation. If the offsets are already gone, I record a gap. I do not turn off failOnDataLoss to skip ahead.

Q7. Why is retention segment-granular, and what does that mean for a privacy delete of one learner?

The broker deletes old segments after they roll and age out. A record in the active segment, or in a closed segment that is not old enough, remains. You cannot promise a single learner's keys disappear at a precise time by setting retention.ms. A targeted delete is a lake rewrite or another design. I will not tell compliance that topic retention is a per-learner erasure control.

Q8. When would you compact learning.enrollment, and why will you not compact progress?

Compaction keeps the latest record per key and drops the rest. Progress needs every observation, so compaction would destroy the tile and the morning rebuild. Enrollment current state is latest-wins if the latest payload carries the delete. I would compact only a topic whose consumers are written for that definition, and I would keep a delete-retention landing in the lake for the audit of intermediate statuses. A null tombstone that removes the key is the wrong delete marker if a rebuild must see op=delete.

Q9. How do you notice a hot partition, and why is adding partitions the wrong fix for a hot key?

Bytes-in and consumer lag per partition show one partition ahead of the others. Topic-level rates hide it. If many distinct keys are too fast, more partitions and more group members can spread the work. If one learner_id is too fast, the hash still selects one partition after the increase. I rate-limit or fix that client. I do not sell a partition increase as a hot-key remedy.

Q10. What do you paste into the topic ticket so lesson 40 and the tile can rely on it?

Partition counts, keys, replication factor, minimum ISR, unclean election off, cleanup policy, retention.ms, segment.ms, and the sentence that landing offsets and the Spark checkpoint must remain inside retention. I also paste the topic id. I do not paste a producer linger.ms. That is a client setting. The ticket is the log's contract, and both the group cursor and the checkpoint are clients of it.

Hands-On Exercises

Exercise: Create the progress topic and break the ISR on purpose

Problem statement. Create learning.progress with the durability and retention settings in this lesson. Show same-key placement. Show produce failure when the minimum ISR cannot be met, or, if you cannot stop brokers, show the effective config and write the failure you must observe later.

Tasks.

1. Create the topic and the DLQ with Admin or kafka-topics.sh. Capture effective configs, not only the flags you passed.
2. Produce two records with the same learner_id and show one partition. Produce a second learner and show that a different partition is allowed.
3. Stop brokers until acks=all fails, or document the lab limit and the exception you expect.
4. Write the retention sentence naming the landing group and the Spark checkpoint.
5. Explain why you did not set cleanup.policy=compact on progress, in five lines a player engineer can read.
6. State what --alter --partitions would do to an existing learner, and why this exercise does not run that command on a shared lab topic other people are reading.
7. Record replication as a cost: three copies of fourteen days, and the rebuild headroom from lesson 37.

Expected deliverables. Config transcript, partition evidence, the ISR test or the written expectation, and the retention paragraph.

Evaluation rubric.

You pass when a reviewer can see min.insync.replicas=2 on the effective config and can point at the partition both records for one learner used.

Lesson Summary

Partitions are how learning.progress scales and where its order stops. Twelve partitions and a learner_id key are the planning start. The default hash is stable only while the count is stable. Adding partitions splits new events for a learner and does not move old ones. Replication factor 3, minimum ISR 2, and acks=all acknowledge a write only when two copies have it. Unclean leader election stays off so offsets do not change identity. Retention of fourteen days, with segment roll on the order of a day, exists so landing and the Spark checkpoint can resume after a real outage. Deletion is segment-granular. Compaction is for latest-per-key state, not for progress observations. The topic id and these configs are the contract lesson 40's group and lesson 32's checkpoint both read.

Kafka Consumer Groups and Offset Management is how landing divides those partitions and commits a position. The primer's Replication, ISR, and Retention Policies pages go further into broker mechanics.

Further Reading

• Topic configurations — retention.ms, segment.ms, min.insync.replicas, cleanup.policy, unclean.leader.election.enable, message.timestamp.type.
• Admin.createTopics — NewTopic and the config map used in the walkthrough.
• Replication, Log Compaction, and Partition Strategy — specialty primer. The bus choices are the table in this lesson.
• Structured Streaming with Spark — failOnDataLoss and the checkpoint this retention has to cover.