Streaming Data Architecture

Data Engineering & Modern Data Platforms · Lesson 36 · Real-Time Data Engineering and Kafka

Lesson Overview

The five-minute course activity tile from Structured Streaming with Spark already reads a topic named learning.progress. The morning course_day table from Spark Performance Optimization is still the historical record, ready at 06:00 UTC, with a restatement window for late event time. The cohort counter from Change Data Capture Fundamentals is a third product: it must notice a hard unenrollment within about fifteen minutes, and a daily snapshot of enrollment is the wrong instrument for that. A design review last week proposed one sentence as the architecture: put progress, enrollment, and the Monday number on Kafka, and retire the batch.

That sentence collapses three clocks and two kinds of fact into one pipe. Progress is append-only. A learner's client emits lesson_progress when a lesson advances. Losing one event loses a step in the sequence; keeping every event is the product. Enrollment is a current-state fact with deletes. The row that matters is the latest committed status for (learner_id, course_id), and a DELETE that never appears downstream overfills a cohort. The Monday completion number is a batch fact. It is allowed to be late relative to a live lesson, and it is not allowed to drop an event because a watermark closed. One log can carry all three payloads. One processing policy cannot serve all three consumers.

This lesson designs the TechLearningPro progress and enrollment event bus as a data product, before any broker count or producer setting. A stream, in this course, is an ordered, replayable history of facts that more than one consumer can read without polling the system of record. A broker is the storage that makes that history durable for a retention window. A producer is a writer with a contract. A consumer is a product with its own cursor, its own freshness bound, and its own tolerance for duplicates and lateness. Kafka is the log this platform standardizes on. The specialty primer teaches the log as a system. This module teaches the log as a stage in the lifecycle: what lands, what is raw, what is curated, which job is allowed to speak for the business, and what a replay is allowed to change.

The bus has two topics in the contract you will carry through lessons 37–41. learning.progress is an append-only event log. learning.enrollment is a change log for enrollment status, shaped so a later CDC lesson can fill it without renaming the downstream grain. Neither topic is the warehouse. The lakehouse raw zone is a consumer of the log. The tile is another consumer. The cohort counter is another. The morning batch reads landed files, not the broker, once landing has succeeded inside retention. If landing falls behind retention, the batch has a hole, and that hole is an incident in this architecture, not a surprise discovered in Module 5.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate a freshness bound from a tool choice, and say which TechLearningPro products justify a continuous log.
2. Distinguish event time from processing time, and place each clock on the progress event and on the morning table.
3. Define producer, consumer, and broker as roles in a pipeline, with a cursor and a retention window attached to each consumer.
4. Choose an append-only log for progress and a change log for enrollment, and keep the Monday completion fact on the batch path.
5. Write a consumer matrix that names the cursor each job owns, including the Structured Streaming checkpoint versus a later group offset.
6. State what replay can reconstruct, what it must not silently overwrite, and which dataset remains the system of record after landing.
7. Apply the six-part decision lens (correctness, freshness, scale, reliability, security, cost) to a proposal to "stream everything."

Prerequisites

• Data Integration Patterns: a log is one movement pattern, chosen when a consumer's freshness bound and the source's ability to emit an ordered change both say so.
• Change Data Capture Fundamentals: enrollment deletes are invisible to a watermark poll. This lesson reserves a change log for that fact. It does not deploy a connector. Lesson 46 does, after the log itself is in place.
• Structured Streaming with Spark: the activity tile already treats learning.progress as a replayable source. Its cursor is a checkpoint. Carry that fact forward unchanged.
• Data Quality Engineering and Data Formats and Serialization: the payload in this module is UTF-8 JSON bytes. A bad record is a quality event, not a reason to block the partition forever.
• ETL and ELT Architecture: landing the log into a raw zone and reducing it later is ELT. Computing the Monday number inside the producer is how you lose the replay.
• /learn/kafka, especially What is Kafka? and Kafka Use Cases, if the words topic and offset are new. This lesson uses them as contract terms. Lessons 37–41 give them operational meaning. The primer remains the place for CLI tours, Spring listeners, and broker internals that a pipeline design does not need yet.

Detailed Core Concepts

A fact, a clock, and a consumer

A streaming architecture starts from the fact, not from the cluster. A fact is something a source system asserts at a moment: the player asserts that learner lrn_1842 reached 40 percent of a lesson; PostgreSQL asserts that the same learner's enrollment row committed with status unenrolled. The clock on the fact is event time, the time the source claims. The clock on the pipeline is processing time, the time a consumer handles the record. They diverge whenever a phone reconnects, a job restarts, or a partition lags.

The consumer is who turns that divergence into a product decision. The activity tile promises a five-minute window and allows ten minutes of lateness, then drops anything older. The morning table promises a restatement window measured in days and is allowed to correct yesterday. The cohort counter promises current status within about fifteen minutes and must apply a delete. Three promises, three consumers, one bus if the facts are worth sharing, and three cursors so a rewind of the tile does not rewind the cohort.

Planning assumptions, labeled as such and not as measured production traffic: progress is the high-volume append stream, bursting while a lesson is live; enrollment changes are orders of magnitude smaller and include deletes; the Monday review reads yesterday. Those assumptions decide the shape. They do not decide a partition count. Lesson 39 does that, after the contract is stable.

Event time and processing time

Event time lives in the payload as event_time, an ISO-8601 instant in UTC, produced by the client or by the database clock that committed the change. Processing time is assigned when a consumer sees the record. Broker append time, which Kafka can stamp as LogAppendTime, is a third clock. It answers "when did the log accept this record?" It does not answer "when did the learner watch the lesson?"

The tile in lesson 32 windows on event_time. A job that windows on processing time will count a train-tunnel backlog as if it happened at reconnect, and the five-minute tile will show a spike that no instructor can explain. The morning batch keys off event time as well, with a longer restatement, so a late client still lands in the right course-day. Using the broker timestamp as a substitute saves a parse and breaks both products the first time a consumer falls an hour behind and then catches up.

Producers set event_time from the device clock or the database clock, and they also send the record promptly. A device clock can be wrong. The contract therefore includes event_time and, once the record is in the log, the broker's append timestamp. Quality rules in lesson 47 can flag device clocks that sit far from append time. This lesson only requires that both clocks exist and that analytics name which one they use. Do not overwrite event_time with processing time "to make the dashboard move."

The log, the queue, and the table

A queue hands a record to one consumer and then forgets it. A table stores the latest value of a key and forgets how it got there, unless you added history on purpose. A log appends facts, keeps them for a retention window, and lets every consumer read at its own position.

TechLearningPro needs the log when two products must see the same progress fact without both polling the player database, and when a consumer must be able to restart from a known position. The activity tile and the raw landing job are those two products today. A third, a quality auditor, can subscribe later without asking the player team for a new feed. That fan-out is the reason the bus exists.

The log is a poor system of record for "who is enrolled right now" if you only keep it for seven days and you never compact it. Current enrollment for the life of the company is a table, fed by the change log. The change log is how the table stays fresh and how you can rebuild it. Lesson 16 already separated the cohort's current-state table from the historical completion fact. This lesson gives the change log a name, learning.enrollment, and leaves the table where the cohort job writes it. Compaction, which can keep the latest record per key inside Kafka itself, is a topic setting in lesson 39. It is an optimization for a consumer that wants the latest status from the log. It is not a substitute for the cohort table the instructors query.

A queue would fit a single worker that sends email on enrollment and must not send it twice from a fan-out. That worker is not this module. If a later product is a true work queue, Kafka vs RabbitMQ, Pulsar & Others is the comparison. The progress bus is a log because the landing files and the tile both need the same history.

Two topics, two fact types

learning.progress carries append-only learning activity. Every accepted event stays meaningful. A later event does not erase an earlier one. The grain of the event is one observation: a learner, a lesson, a progress percent, an event time, and an event_id the player generated. The ordering that matters is per learner, so that 40 percent followed by 55 percent stays in that order for that learner. Ordering across learners does not matter. The analytics grain of the tile is course_id plus a window. Those are different keys. Lesson 39 will refuse to partition by course_id just because the tile groups by it. A hot course would pin a partition, and learners in that course would share a single ordered queue they do not need.

learning.enrollment carries changes to enrollment status. The grain of the change is one committed modification of (learner_id, course_id): an upsert or a delete, with the source log position when the source is PostgreSQL (source_lsn from lesson 16). Applying changes in source order yields current status. Applying them out of order can resurrect a deleted enrollment. The historical completion mart does not read this topic to rewrite yesterday. A delete today must not remove last month's completion. That rule survived lesson 16 and it survives the move onto a log.

Putting both fact types on one topic, distinguished only by a field inside JSON, forces every consumer to filter, complicates retention (you cannot compact one event type and delete the other on a shared key space without pain), and makes a poison progress record stall an enrollment consumer that shares the partition. Two topics cost an extra ACL and an extra dashboard. They keep the failure domains apart. Schema Registry and a formal compatibility policy are lesson 43. Until then the contract is a version field and a written table of required keys, and producers send UTF-8 JSON bytes.

Consumers are products with cursors

A consumer group, taught in lesson 40, is how several instances of one job divide partitions. It is not how several products divide the log. Each product has its own group, or, in the tile's case, no group cursor at all.

The tile's checkpoint and the landing job's group offset move independently. Rewinding the landing job to repair a bad file must not move the tile. Lesson 32 already said the group offset is the wrong recovery cursor for Spark. This table is where that sentence becomes an architecture.

Lambda, kappa, and the choice on this platform

Lambda architecture runs a batch path and a speed path and tries to reconcile them. Kappa architecture treats the log as the only path and rebuilds tables by replaying it. Both names are useful as warnings. Neither is a mandate.

The speed path is the tile and the cohort counter. The batch path is course_day and the completion mart. They share landed progress events and do not share a definition of done. The tile drops late event time. The batch restates it. Summing tiles into the morning table under-counts. Replay within retention can rebuild a derived dataset. Eighteen months of completions cannot depend on a fourteen-day topic. The lakehouse is the long copy after landing. The log is the short, ordered, fan-out copy.

Replay and the system of record

Replay means reading the log again from an offset you choose, within retention. It reconstructs what the log still holds. It does not reconstruct what retention deleted, and it does not undo a table write that a consumer already committed unless that write is idempotent.

After the landing job has written a durable object and committed the offset, the raw object is the system of record for that event in the analytics platform. The log remains the system of record for consumers that have not landed yet, and for a replay that the raw zone lost. Two systems of record sound like a contradiction. They are a handoff. The handoff is explicit: an object exists, its manifest lists the offsets it contains, and the group offset is at or ahead of that range. If the object exists and the offset commit failed, the next run writes the same offsets again and the object identity must make that a no-op. Lesson 41 specifies the delivery semantics. This lesson only requires the handoff to be named, so a runbook does not "start from latest" and call the hole a feature.

When a stream is the wrong tool

A stream is the wrong tool when the consumer's bound is the next morning, the source can be extracted without hurting the primary, and no second consumer needs the feed. Lesson 16 kept the daily enrollment snapshot for the completion join for that reason. Moving that snapshot onto learning.enrollment "for consistency" adds a live consumer, a retention obligation, and a delete-ordering rule to a product the batch already finishes by 06:00 UTC.

A stream is also the wrong tool when the source cannot emit an ordered fact. A nightly file from a partner is a file. Landing it on a topic one row at a time adds a broker between two batch jobs and creates a cursor nobody will rewind correctly. API and File-Based Data Ingestion still owns that path.

The stream is the right tool for learning.progress because the tile's bound is minutes, the landing job wants the same bytes the tile saw, and polling the player database at that rate is a load the player team has already refused. The stream is the right tool for enrollment changes because the cohort bound is about fifteen minutes and the fact includes deletes. Those are the two justifications. Freshness language in a ticket is not a third justification until it names the bound and the source.

The decision lens on the bus

Correctness is every accepted event_id in the raw zone, enrollment state in source order including deletes, and a Monday table that does not inherit the tile's watermark. Freshness is minutes for the tile, about fifteen minutes for the cohort, and 06:00 UTC for course_day. Produce latency is the small term. Consumer lag is the rest. Scale follows progress bytes. Enrollment is the smaller topic. Reliability is a cursor per consumer, retention that covers the replay you intend, and a quarantined poison record so the partition continues. Security is a player principal that may append to learning.progress and may not read enrollment, a landing principal that may read progress and append to quarantine, and the same boundary on the raw zone. ACLs are the control list. Cost is broker disk for the retention window, lake objects for as long as you keep them, and people who can explain two cursors. Cutting retention to save disk, without measuring landing lag, is a correctness incident.

Architecture and Design Diagrams

mermaidCopy
flowchart LR
    subgraph sources [Sources of fact]
        player[Player and lesson client]
        pg[(PostgreSQL enrollment)]
    end

    subgraph bus [Progress and enrollment bus]
        progress[learning.progress append log]
        enroll[learning.enrollment change log]
    end

    subgraph consumers [Products and their cursors]
        land[Raw landing group offset]
        tile[Activity tile checkpoint]
        audit[Quality auditor group offset]
        cohort[Cohort table group offset]
    end

    subgraph record [Systems of record after handoff]
        raw[(Lakehouse raw zone)]
        day[course_day batch table]
        now[Cohort current-state table]
    end

    player -->|event_time and event_id| progress
    pg -->|committed change and source_lsn| enroll
    progress --> land
    progress --> tile
    progress --> audit
    enroll --> cohort
    land --> raw
    raw --> day
    cohort --> now

The player does not write the morning table. The morning job does not read the broker once the raw object for that offset range is durable. The cohort job does not read learning.progress to decide who is enrolled.

mermaidCopy
sequenceDiagram
    participant Player
    participant Log as learning.progress
    participant Landing
    participant Raw as Raw object store
    participant Tile as Structured Streaming

    Player->>Log: append JSON bytes keyed by learner_id
    Note over Log: offset assigned on one partition
    Landing->>Log: fetch from group offset
    Landing->>Raw: write object named by topic partition offset range
    Landing->>Log: commit group offset after durable write
    Tile->>Log: fetch a trigger range
    Note over Tile: checkpoint records the range, not the group offset
    Tile->>Tile: watermark on event_time, append closed windows

A third diagram is the decision, not the deployment:

mermaidCopy
flowchart TD
    q1{Consumer bound inside minutes or a short hour?}
    q2{Source can emit an ordered fact, including deletes if deletes matter?}
    q3{More than one product needs that fact, or polling the source is unacceptable?}
    batch[Batch extract or file landing]
    log[Place the fact on the bus and name the cursor]
    q1 -->|No| batch
    q1 -->|Yes| q2
    q2 -->|No| batch
    q2 -->|Yes| q3
    q3 -->|No| batch
    q3 -->|Yes| log

Walk a proposal through this before you open a topic ticket. The tile passes all three. The partner's nightly seat file fails the second and stays a file. The daily enrollment snapshot for the completion mart fails the first and stays a snapshot. The cohort counter passes all three and gets learning.enrollment.

Technical Explanation

What "ordered" is allowed to mean

Order is a property of a single partition of a log, not of the bus. Across partitions, records are concurrent. Any design that says "events are totally ordered on Kafka" is false, and any dashboard that depends on total order will flicker. The contract orders a learner's progress by partitioning on learner_id and by carrying event_time and a monotonic event_id from the player. If the player retries and appends the same event_id twice, those two records are both in the log. Order of distinct observations is preserved per learner. Uniqueness is a consumer concern, stored with event_id, and it is specified in lesson 41. Stating it here stops a producer design that assumes the broker will collapse retries of a new process.

Enrollment order is stricter. The source order is the PostgreSQL LSN, not the order in which a publisher happened to flush. The change record carries source_lsn. The cohort consumer applies a change only when its LSN is newer than the LSN stored for that key. A publisher that sends two changes out of LSN order must not resurrect a row. That rule is lesson 16's fold, moved next to a log. The log's partition order should still follow the key so that one key's changes share a partition. The LSN remains the authority if a publisher bug reorders them.

The envelope

Every record on both topics is a UTF-8 JSON object in the value bytes. The key is a string, not JSON. Progress keys are learner_id. Enrollment keys are learner_id concatenated with a unit separator and course_id, so one enrollment shares a partition and compaction, later, can keep the latest status for that pair. The value carries the same identifiers again. Consumers must not trust the key alone for analytics, because a buggy producer can mismatch them. A quality check compares key and payload and quarantines mismatches.

Required progress fields: event_id, event_type, event_version, event_time, learner_id, course_id, lesson_id, progress_pct, producer. Required enrollment fields: event_id, event_type, event_version, event_time, learner_id, course_id, op (upsert or delete), status (null on delete), source_lsn. event_version starts at 1. Lesson 43 will decide how a compatible field is added. Until then, consumers ignore unknown fields and reject missing required fields.

progress_pct is an integer from 0 to 100. It is not a float the player computed with binary fractions. event_time must parse as UTC. A timestamp without a zone is a quarantine, not a guess of America/Los_Angeles. Null learner_id is a quarantine. These rules are the same rules a batch file would have faced in lesson 17. Streaming changes when they run, which is on the landing and auditor path continuously, and it does not retire them.

Raw landing as ELT

The landing consumer writes JSON Lines to the raw zone, one object per record, partitioned by event date taken from event_time only after the record validates. Invalid records do not land in the dated progress prefix. They land on a quarantine topic, learning.progress.dlq, with the original bytes preserved and a header that names the reason. The raw prefix is then safe for the morning job to read without re-implementing the parser's failure modes. This is ELT: the raw zone holds accepted facts, the curated course_day is a later reduction, and the transformation is SQL or Spark in the batch window, not hidden inside the player.

Each raw line includes the log coordinates the consumer observed: topic, partition, offset, and the broker timestamp. Those coordinates are the idempotency key for the write. A replay of the same offset replaces the same line. A second event_id at a new offset is a new line, and the curator can dedupe if the product definition says a retried player event is one fact. The landing job does not make that product decision silently by dropping the second offset. Dropping it would hide a producer bug. Keeping it, with event_id visible, lets the morning model decide.

Failure domains

Four failures stay independent in the runbook. A producer that fails before an acknowledgement retries with the same event_id, and the log may hold two records. A broker loss must still leave an acknowledged progress event, which lesson 39 turns into replication factor, minimum in-sync replicas, and acks. A landing crash after the object write and before the offset commit repeats the write, so the object identity has to make that a no-op. Committing before the write is at-most-once and is the wrong order. The tile resumes from its checkpoint, which may sit behind or ahead of the landing offset. Resetting both cursors to the same number couples two products that are supposed to diverge.

What the later lessons are allowed to assume

Lessons 37 through 41 may assume the topic names, the keys, the JSON envelope, the consumer matrix, and the requirement that an acknowledged progress event survives one broker loss. They may not reopen the decision to keep course_day on the batch path. Lesson 38 uses org.apache.kafka.clients and does not add a method named sendExactlyOnce. Lesson 39 sets retention and partition counts. Lesson 40 commits after a durable side effect and leaves the Spark checkpoint alone. Lesson 41 may claim at-least-once landing plus idempotent writes, the tile's file-sink claim from lesson 32, and a Kafka transaction that does not commit PostgreSQL. A new required field is a contract change in this lesson, with a version bump. A partition count is not.

Practical Implementation or Design Walkthrough

Problem

Specify the progress and enrollment bus so a player team can emit, a landing job can persist, and the tile can keep the checkpoint it already has. No broker is required to finish this lesson. A broker without this contract is how the platform gets a topic named events and three incompatible JSON shapes by Friday.

Source and target

Source 1 is the lesson player. It can emit an HTTPS call when progress advances. It cannot offer a SQL extract at tile latency. Source 2 is PostgreSQL enrollment, whose change semantics are lesson 16. The publisher of those changes is out of scope until lesson 46; the record shape is in scope now so the cohort job is not rewritten when the publisher arrives.

Targets: raw JSON Lines for progress, a quarantine topic for rejects, a checkpointed tile that already exists, and a cohort table that applies enrollment changes. The morning course_day job remains a batch reader of the raw zone.

Schema and sample records

Progress value:

jsonCopy
{
  "event_id": "9f1c0d5e-6a2b-4e7a-9c11-0b2e6d4a1180",
  "event_type": "lesson_progress",
  "event_version": 1,
  "event_time": "2026-11-11T00:41:12.184Z",
  "learner_id": "lrn_1842",
  "course_id": "de-modern-platforms",
  "lesson_id": "structured-streaming-with-spark",
  "progress_pct": 40,
  "producer": "web-player"
}

The record key is lrn_1842, not the JSON object.

Enrollment value:

jsonCopy
{
  "event_id": "2c44b1aa-11e0-4d5f-8a77-6e0b9c1d4402",
  "event_type": "enrollment_changed",
  "event_version": 1,
  "event_time": "2026-11-11T00:40:01.002Z",
  "learner_id": "lrn_1842",
  "course_id": "de-modern-platforms",
  "op": "delete",
  "status": null,
  "source_lsn": "0/16B3748",
  "producer": "enrollment-publisher"
}

The record key is lrn_1842 plus a unit separator plus de-modern-platforms.

A validator the landing job can share with tests

The function below is application code. It does not talk to a broker. It classifies one JSON object. The landing loop in lesson 38 calls the same rules after it reads bytes. Rejects include a stable reason so the quarantine topic is queryable.

pythonCopy
"""Progress envelope checks. Python 3.11. Standard library only."""

from __future__ import annotations

import json
from dataclasses import dataclass
from datetime import datetime
from typing import Any

REQUIRED = (
    "event_id",
    "event_type",
    "event_version",
    "event_time",
    "learner_id",
    "course_id",
    "lesson_id",
    "progress_pct",
    "producer",
)


@dataclass(frozen=True)
class Decision:
    accept: bool
    reason: str
    event_time: datetime | None = None


def classify_progress(raw: bytes, key: str | None) -> Decision:
    try:
        payload: Any = json.loads(raw.decode("utf-8"))
    except (UnicodeDecodeError, json.JSONDecodeError):
        return Decision(False, "unparseable")
    if not isinstance(payload, dict):
        return Decision(False, "not_an_object")
    missing = [name for name in REQUIRED if payload.get(name) is None]
    if missing:
        return Decision(False, "missing:" + ",".join(missing))
    if payload["event_type"] != "lesson_progress" or payload["event_version"] != 1:
        return Decision(False, "unsupported_type_or_version")
    if key is not None and key != payload["learner_id"]:
        return Decision(False, "key_payload_mismatch")
    try:
        event_time = datetime.fromisoformat(payload["event_time"].replace("Z", "+00:00"))
    except (TypeError, ValueError):
        return Decision(False, "bad_event_time")
    if event_time.utcoffset() is None:
        return Decision(False, "event_time_without_zone")
    pct = payload["progress_pct"]
    if not isinstance(pct, int) or isinstance(pct, bool) or not 0 <= pct <= 100:
        return Decision(False, "bad_progress_pct")
    return Decision(True, "ok", event_time)

bool is rejected because a JSON true must not count as progress 1. Integer check uses Python's subclass relationship on purpose. The test file that ships with the exercise covers a missing field, a naive timestamp, a mismatched key, progress_pct of 140, and a valid sample. Those five cases are the contract. A producer sample that passes them is allowed to request credentials. A producer sample that fails them does not get a topic exception "just for the pilot."

Consumer cursors in the runbook

Write this into the service catalog next to the topic names.

• Landing group id: learning-progress-raw-landing. Commit only after the raw object and its manifest are durable. Manifest lists topic, partition, inclusive offset start, inclusive offset end.
• Tile: checkpoint path owned by the Spark job in lesson 32. No group id is the recovery cursor. A group id may appear in broker metrics if a source library creates one; operators still resume the checkpoint.
• Auditor group id: learning-progress-quality. It may lag the landing job. It must not commit the landing group's offsets.
• Enrollment group id: learning-enrollment-cohort. Reserved until the publisher exists. Creating the empty topic now is optional; inventing a second JSON shape later is not.

Validation and the morning handoff

A raw partition for a date is ready for course_day when landing has committed offsets through the broker end offset as of 02:00 UTC, or when the runbook records an intentional gap. The batch job prints the max offset it observed per partition in the objects it read. That number is how you notice landing stopped at 18:00 and the batch still published a green run. Freshness of the batch is 06:00. Completeness of the batch is the offset handoff. Both belong on the job's success condition.

Deployment notes

The player team owns producer values and event_id generation. Platform engineering owns topic configuration, ACLs, and retention, specified in lessons 37 and 39 from the requirements in this contract. Data engineering owns the landing job, the validator, the quarantine reason list, and the handoff check in the morning job. Nobody owns "the Kafka pipeline" as a single on-call blob. An incident names the cursor that stopped.

Dependencies for this lesson are Python 3.11 and the contract tables. The official client appears in lesson 38: org.apache.kafka:kafka-clients, classes KafkaProducer and KafkaConsumer. Do not add a Python binding in this lesson and call it the implementation. The validator is the implementation of the contract. The client is the implementation of the transport.

Enterprise Real-World Scenarios

Scenario 1: The live tile and the Monday number

Product asks for the activity tile to "match finance." Finance means course_day, which restates three days of late event time and counts distinct completed lessons under a definition the tile does not implement. The tile counts progress events in a five-minute window and drops rows older than the watermark.

The architecture keeps both. The tile subscribes to learning.progress and writes Parquet with the checkpoint from lesson 32. Finance reads course_day, built from raw landing. A reconciliation job may compare a closed hour on a quiet day to catch a landing bug. A difference equal to late events is expected and is labeled late, not failed. A difference that grows after the restatement window closes is an incident.

The tile principal reads the topic and writes the tile path. Finance reads the mart and has no producer ACL. The tile's state is windows times courses. The log's volume is events times retention. Those are different capacity plans.

Scenario 2: The cohort counter meets a delete

Support deletes a test enrollment. Lesson 16 showed why a poll on updated_at misses it. The bus carries an enrollment_changed record with op equal to delete and a source_lsn. The cohort consumer folds it into the current-state table and the seat count drops within the fifteen-minute bound. The completion mart for last month still shows the learner, because that mart is built from the daily snapshot history, not from current state.

A second proposal says to emit only the new seat count from the application and skip the change record. The seat count cannot be replayed into a different cohort definition, and a missed publish cannot be rebuilt from PostgreSQL without going back to lesson 16's log. The change record is the fact. The seat count is a query. Publish the fact.

If the publisher is down, the cohort lags and the replication slot from lesson 16 still retains WAL. Lag on the group is the page. The slot monitor stays until that publisher is the only reader of the slot.

Scenario 3: A player bug emits progress percent 140

The validator quarantines the record and landing continues. The tile's parser has to apply the same range check, or the window and the morning job diverge. Shared tests, imported by the player and by landing, are the contract. The quarantine topic keeps the bytes so the bad hour has a sample. A silent drop leaves a short Monday number and nothing to inspect.

Best Practices

• Name the fact, the event-time field, the key, and the consumer's cursor before you name the cluster. A topic without those four is a bucket.
• One product, one cursor. The landing group, the auditor group, and the Spark checkpoint are different objects even when they read the same partitions.
• Keep append-only observations and current-state changes on separate topics so retention and failure domains can differ.
• Put event_id in the payload at the source. The broker's offset identifies a log record. It does not identify a player retry that became a second log record.
• Land first, then let batch curate. The producer publishes facts. It does not publish the Monday aggregate.
• Quarantine with a reason and the original bytes. Continue the partition.
• Size retention to the slowest replay you intend to support, and write that number next to the topic. Disk alerts are not a retention policy.
• Treat learner identifiers as sensitive in the log and in the raw zone. Restrict producer and consumer principals to the topics they need.
• Version the envelope with event_version. Ignoring unknown fields is the compatibility rule until lesson 43 replaces it with a registry.
• When a proposal says "real time," rewrite it as a bound in minutes and walk the decision diagram. Bounds that are "next morning" stay on the batch path.

Common Mistakes and Anti-Patterns

One topic named events. Progress, enrollment, clicks, and heartbeats share a partition and a retention clock. A bad heartbeat schema stalls enrollment or forces every consumer to deserialize every record. Split by fact type when order, retention, or ACL boundaries differ. A shared topic is justified only when the facts are one product and one schema version.

Streaming the Monday number because the tile exists. The tile's watermark is a different definition. Publishing course_day from a stream job that drops late data makes finance wrong and makes the batch job look redundant until audit. Keep the batch. Point the stream at the product whose bound is minutes.

Using the broker timestamp as event time. Catch-up after an outage rewrites history into the hour of the catch-up. Store source event time in the payload and window on that. Keep the broker timestamp as an operational column.

Committing the cursor before the side effect is durable. The group moves on, the object store has a hole, and retention deletes the only copy. Commit after the durable write. Make the write idempotent so the crash between write and commit is safe.

Resetting every cursor to the same offset during an incident. The tile and the landing job are supposed to diverge. A shared reset couples their bugs. Move the cursor that lost data, and leave the healthy one.

Partitioning the log by the analytics grain. course_id matches the tile's groupBy and concentrates a popular course on one partition. The order you need is per learner. Key by learner_id. The tile shuffles by course_id inside Spark, which lesson 30 already taught you to watch.

Treating retention as the warehouse. Fourteen days on the broker cannot answer an eighteen-month completion question. Land to the lake, and let the broker serve fan-out and short replay.

Dropping poison records on the floor. The partition stays healthy and the Monday number quietly shrinks. Quarantine is the design. A consumer that throws and stops is preferable to a consumer that skips without a record, and a consumer that quarantines is preferable to both for a known validation failure.

Troubleshooting and Design Review

Symptom: The tile shows a burst at 01:10 UTC. Instructors say the lesson ended at 23:00. The morning table, after restatement, places the same event_id values on the previous course-day.

Likely cause: a consumer or a dashboard used processing time or broker append time. The client's event_time was 23:00. The phone uploaded the batch at 01:10.

Investigation: compare event_time in the payload to the broker timestamp on the same offset. If they differ by the length of the outage, the log is fine and the query is on the wrong clock.

Corrective action: point the tile and the dashboard at event_time. Leave the broker timestamp in the raw line for lag analysis. Do not rewrite event_time to match the burst.

Symptom: course_day for yesterday is short by a few percent. Landing lag is near zero at 06:00. The quarantine topic grew at 19:00.

Likely cause: a player release sent progress_pct as a float or omitted event_time's zone. The validator did what the contract says.

Investigation: group the quarantine topic by reason and producer. Pull one raw payload. Compare it with the sample in this lesson.

Corrective action: fix the player, replay from quarantine only if the repaired records can be given valid fields without inventing event time. If event time was absent, you cannot invent it from processing time and still call the table event-time correct. Report the gap.

Symptom: After a landing deploy, the tile's counts doubled for one hour and the raw zone also has duplicate objects.

Likely cause: an operator reset the Spark checkpoint and the landing group to earliest, and the landing write is not idempotent on offset.

Investigation: checkpoint history, group offsets, and whether object names include the offset range.

Corrective action: stop both jobs. Decide the intended start per cursor. Repair files by rewriting objects for the affected offset ranges. Resume the tile from the checkpoint that matches the last good sink commit if it still exists. A shared earliest reset is not a repair.

Symptom: The cohort counter still shows a deleted test learner. learning.enrollment has no record. The PostgreSQL slot lag is growing.

Likely cause: the publisher is down or pointed at the wrong table. The bus is empty because the source change never arrived. The slot, not Kafka, is retaining WAL.

Investigation: slot restart LSN from lesson 16, publisher process health, and end offset of learning.enrollment.

Corrective action: restore the publisher. Do not invent a delete on the topic from memory. If the slot still holds the delete, the publisher can emit it with the original LSN. If the slot was dropped and the WAL is gone, take a new snapshot and record the discontinuity. The cohort table is rebuilt from the snapshot plus the new stream, not from a hand-written event.

Symptom: A design review cannot say who is allowed to rewind learning.progress.

Likely cause: the topic was created without the consumer matrix.

Investigation: list principals and group ids. If you find one shared group id used by landing and by an ad-hoc notebook, you have found a future incident.

Corrective action: split group ids, remove the notebook's write access to the landing group, and document rewind authority. Read access for debugging can use a throwaway group that starts at latest and commits nothing that production reads.

A design review of this architecture asks seven questions. What is the fact on each topic? What is the key, and what order does that key preserve? What is each consumer's freshness bound? Which cursor does that consumer own? What is the system of record after a successful handoff? How long must the log retain data for the slowest of those consumers? Which product remains on batch, and why is that not a failure to modernize? A review that answers with a broker count has not finished the review.

Interview Questions and Answers

Q1. What is a streaming architecture deciding, before it selects a broker?

It decides which facts move continuously, which clocks those facts carry, which consumers read them, what freshness each consumer promises, and which dataset remains the historical record. On this platform the continuous facts are lesson progress and enrollment changes. The historical completion number stays a batch table built from landed progress. The broker is how the continuous facts are retained and fan out. Choosing Kafka in the first sentence skips the products that should not be on the log.

Q2. Why do event time and processing time both need to exist on learning.progress?

Event time is when the learner advanced, and it is the clock for the five-minute tile and for course_day. Processing time, and the broker append time next to it, is when the pipeline saw the bytes. Clients delay uploads. If you window on processing time, a catch-up looks like a live burst and yesterday's course-day loses the events. If you throw away append time, you cannot see lag or a broken device clock. Store both. Aggregate on event time. Operate on the difference.

Q3. How is a log different from a queue and from a current-state table for this bus?

A queue delivers a record to one worker and drops it. A table keeps the latest value of a key. The progress bus needs many products to read the same observations, and it needs a restart position. That is a log with retention. Current enrollment is still a table the cohort job maintains, because instructors query "who is enrolled now" for the life of the course, longer than topic retention. The log feeds the table. It does not replace it. A queue would fit a single email worker, which is a different product.

Q4. Why are progress and enrollment two topics?

Progress is append-only, high volume, and keyed for per-learner order. Enrollment is lower volume, includes deletes, and is ordered by source LSN for a learner-course pair. They have different retention needs, different ACLs, and different poison-record blast radii. A shared topic couples those choices and makes every consumer parse both shapes. The cost of a second topic is small next to a stalled cohort caused by a bad progress payload.

Q5. What does the consumer matrix prevent during an incident?

It prevents one cursor from standing in for another. Landing commits a group offset after a durable object write. The tile commits a Spark checkpoint. The auditor has a third offset. Resetting "the consumer" is not an operation this platform has. The matrix names the group ids and the checkpoint path so the person on call moves only the cursor that lost data. It also stops a notebook from joining the landing group and committing offsets it did not land.

Q6. When would you refuse to put a TechLearningPro dataset on this bus?

When the consumer can wait until morning, or the source is a file, or only one batch job needs the extract and the source can provide it cheaply. The daily enrollment snapshot for the completion mart is the example from lesson 16. A partner seat file is the example from file ingestion. I would also refuse a proposal whose only justification is that the tile already uses Kafka. A second product needs its own bound.

Q7. What is the handoff between the log and the lakehouse, and who is the system of record?

Landing writes an object that names the offsets it contains, then commits the group offset. After that commit, the raw object is what the morning job reads and what we retain beyond topic retention. The log remains authoritative for consumers that have not landed those offsets, and for a replay if the object is lost and retention still holds the range. A commit without an object is data loss once retention passes. An object without a commit is a duplicate write on restart, which an offset-named object absorbs.

Q8. How do you explain lambda and kappa without picking a slogan?

The platform already has a speed path and a batch path, which is the useful part of lambda: the tile and the morning table answer different questions. The useful part of kappa is replay: within retention, we can rebuild a derived dataset from the log with one definition. We do that for repairs inside the retention window. We do not pretend fourteen days of log can rebuild eighteen months of completions. The lake holds the long copy. The log holds the ordered, fan-out copy. I would describe that design in those sentences and skip the slogan unless an interviewer asks for the label.

Q9. A stakeholder wants the producer to emit the daily completion row directly. What do you protect by refusing?

You protect replay and every other consumer. A completion row is a derived definition. When the definition changes, the events are gone if you never published them. The player also becomes responsible for a warehouse grain, a late-event policy, and a backfill it cannot see. Publish lesson_progress. Derive course_day in the platform, where restatement is already the batch contract.

Q10. What will you hand lesson 37 as fixed, and what will you leave open?

Fixed: topic names, JSON version 1 fields, keys, the rule that event time is the analytics clock, the consumer matrix and cursors, quarantine with reasons, and the requirement that an acknowledged write survives one broker loss. Open: broker count, controller layout, partition count, replication numbers, and the client calls. Those are the next lessons. I will not let a cluster diagram reopen the decision to keep course_day on batch.

Hands-On Exercises

Exercise: The bus contract and the decision diagram

Problem statement. A product manager wants three feeds on the progress bus: lesson progress for the live tile, enrollment deletes for the cohort counter, and a nightly partner file of seat grants that arrives as CSV at 01:00 UTC. Write the contract for the feeds you accept, reject the feed that does not belong, and prove the validator.

Tasks.

1. Walk each feed through the decision diagram. Record the freshness bound, the source of order, and whether a second consumer exists.
2. For the feeds you accept, define the topic name, the record key, required JSON fields, and event-time clock.
3. Implement classify_progress as specified, plus a sibling for enrollment that requires op and source_lsn and allows a null status only when op is delete.
4. Add tests for the five progress cases in the walkthrough and for an enrollment delete, an enrollment upsert missing source_lsn, and a delete that still carries a status.
5. Draw the consumer matrix, including the Spark checkpoint and a landing group, and state the commit order for landing.
6. State the system of record after a successful landing handoff, and the retention question lesson 39 must answer. You do not choose the retention value yet. You write the consumers the value must cover.
7. Explain, in one paragraph, why the partner CSV stays on the file path from lesson 15, and what would have to change before you would revisit that decision.
8. Note the security boundary: which principal produces progress, which principal reads it, and why the partner file's credentials do not belong on the bus ACL.

Expected deliverables. The decision table, the validator module, the tests and their output, the consumer matrix, and the two short paragraphs.

Evaluation rubric.

You pass when a reviewer can implement a producer from your table without choosing a new key, and when the partner file is absent from the topic list for a reason a stakeholder can read.

Lesson Summary

The TechLearningPro bus is two logs with different facts: append-only learning.progress, and learning.enrollment changes that can delete. Event time stays in the payload and is the clock for the tile and for course_day. Processing time and broker append time stay operational. Each product owns a cursor. The landing group commits after a durable, offset-identified object. The activity tile keeps its checkpoint. The morning table reads the lake and restates late event time. The cohort table applies enrollment changes in source order and does not rewrite completion history. A log is justified by a short freshness bound, an ordered source, and a consumer that should not poll that source. A nightly file and a next-morning extract stay off the bus. Replay works inside retention. Long history lives in the lake after the handoff.

Apache Kafka Architecture places this contract on brokers, partitions, and replicas. The specialty primer for the log as a product remains Kafka.

Further Reading

• Apache Kafka documentation, design and introduction — the log, topics, and producers and consumers as the project defines them. Lessons 37–41 use this vocabulary with the contract above.
• Kafka Use Cases and What is Kafka? — the specialty primer. Use it for the mechanism tour. Use this lesson for which TechLearningPro fact belongs on the bus.
• Structured Streaming with Spark — the checkpoint cursor the matrix refuses to merge with a group offset.
• Change Data Capture Fundamentals — LSN order and deletes, which learning.enrollment carries forward.
• Data Integration Patterns — the movement-pattern decision this lesson specializes for a short freshness bound.
• Martin Kleppmann, Making Sense of Stream Processing (O'Reilly) — log-as-integration versus dual writes, at the altitude of this lesson. Broker configuration stays in the official docs and in the next four lessons.