Real-Time Data Quality and Monitoring

Data Engineering & Modern Data Platforms · Lesson 47 · Real-Time Data Engineering and Kafka

Lesson Overview

The pipelines from this module can all look healthy while the product is wrong. The purchase S3 sink from Kafka Connect and Data Integration stays RUNNING and writes objects, while its dead-letter topic absorbs a converter failure and the raw prefix is short. The Spark tile from Structured Streaming with Spark shows no consumer-group lag, because it does not commit a group offset, and someone closes the page because a lag query returned nothing. The enrollment slot from CDC with Kafka and Debezium is inactive, retained WAL is climbing, and the cohort dashboard still shows a snapshot from twelve minutes ago, inside a fifteen-minute promise that is about to become a disk incident.

Data Quality Engineering already defined checks, severities, and the curated publish gate. Those checks stay. They run before a batch publish, on a set, with a quarantine cap. Streaming adds a different clock and a different place to fail. A record goes bad one at a time: a schema the registry rejected, a missing event_time, a sink that could not write, a slot that stopped confirming. The response is a dead-letter topic, a stopped task, or a page. It is not a second copy of the morning gate pasted into a consumer loop. Freshness is a service level objective with a formula and a timestamp field, not a panel of messages per second with no owner.

This lesson defines the streaming checks and the cursors you page on. It does not invent a quality product. It uses the Connect REST status, kafka-consumer-groups.sh, the replication-slot query, and the dropped-record behavior from Event Time, Windows and Late Data. It hands Module 5 a raw prefix, a dead-letter topic, and a freshness number. Data Lakehouse Architecture owns the table commit that comes after the prefix. Topic lag is not that commit. Data Observability and Operational Monitoring widens this into the whole estate. Here the scope is the live path: progress, purchase lines, and enrollment.

/learn/kafka documents broker metrics and the consumer lag column. Those are the measurements. The judgment of which measurement may page a human sits in this lesson.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate record lag, time lag, and event-time freshness, and attach each one to the cursor that actually moves.
2. Explain why a Spark Structured Streaming query does not appear as a caught-up consumer group.
3. Use connector status, sink consumer lag, and dead-letter rate as three different signals.
4. Treat an inactive replication slot's retained WAL as a primary-disk page, distinct from cohort freshness.
5. Assign fail, quarantine, and warn to streaming defects without replacing the batch quality gate.
6. Write a freshness SLO for the activity tile and for the cohort that names the timestamp it uses.
7. Keep schema-registry rejections, late events, and poison clocks in different queues with different owners.
8. Hand the lakehouse a landing contract: prefix, event time, dead letters, and the statement that object creation is not table freshness.
9. Avoid paging on a raw spike during enrollment week, using the same judgment lesson 17 used for volume checks.
10. Review a dashboard that cannot hide a skipped purchase line or a stopped slot behind a green task state.

Prerequisites

• Lesson 17: fail, quarantine, and warn; freshness versus correctness; the curated gate that still runs in the morning.
• Lesson 32: checkpoint offsets, watermarks, and failOnDataLoss. The tile's cursor is not a group.
• Lesson 42: sink dead-letter topics, source errors.tolerance=none, and the raw prefix.
• Lesson 43: a registry incompatibility response is a rejected deploy, not lag.
• Lesson 45: grace drops late events on purpose. A non-zero drop rate can be healthy.
• Lesson 46: slot, Connect offset, and consumer group. Retained WAL is the metric that protects the primary.
• /learn/kafka for consumer-group lag as log end offset minus committed offset. This lesson decides which groups matter.

Detailed Core Concepts

Three different lags

Record lag is the log end offset minus the consumer's committed offset, per partition. Sum it only when a single number is unavoidable, and keep the per-partition view for debugging. kafka-consumer-groups.sh --describe --group GROUP prints it as LAG. It is a count of records, not a duration. A lag of 50,000 can be two seconds of the progress topic or an hour of purchase lines. The command answers how far this group is behind the end of the log.

Time lag estimates how long ago the broker appended the record now at the consumer's offset, usually by comparing timestamps. Kafka does not ship one official "time lag SLO" on the broker. Exporters derive one from record timestamps. If you show it, label the timestamp source as CreateTime or LogAppendTime. Do not label it event-time freshness.

Event-time freshness is now minus the maximum event_time among records successfully applied to the product you are measuring. For the activity tile, the promise is about closed windows, not about the newest raw event. For the cohort, lesson 46 measured from the database commit time on the change event, source.ts_ms, not from when the connector happened to process it. A pipeline can have record lag near zero and poor event-time freshness when clients were offline. It can have large record lag while the prefix already applied is still fresh, with the unread tail about to break the SLO. The SLO names which of these the user feels.

A dashboard that shows only record lag misses the offline-client case and misses the Spark job entirely.

Which cursor the number comes from

Zero lag on a group that has no members can mean the group does not exist. Confirm the group id before you silence a page. The Spark tile looking healthy because a group list had nothing to say is the empty-group mistake.

Streaming checks are not the morning gate

Lesson 17's severities still apply. The place they apply on the live path is narrower.

Fail the task when continuing would skip evidence. JDBC source conversion errors, with errors.tolerance left at none. Debezium task errors. A Streams deserializer failure on purchase lines. Spark failOnDataLoss when the checkpoint points at an offset retention has deleted. These stop progress on purpose. The page says stopped, and the cursor does not move past the bad record.

Quarantine the record when the rest of the stream is still evidence. The sink dead-letter topic for purchase lines and for progress landing. A reject topic for progress events with a missing or future event_time, if you chose a reject path instead of failing the extractor. Quarantine without a count is a drop. The check is the rate and the age of the oldest dead-letter record.

Warn when the definition expects the condition sometimes. Late events past grace, up to a rate the tile's contract allows. A volume dip in the middle of the night. A missing course dimension flag on the revenue topic while the catalog lags. Warns need a reader and a review date, the same rule as lesson 17. An open-ended warn on dead letters is how the prefix went short.

Do not stream the curated gate. Uniqueness of event_id across a day, referential integrity to a pinned lesson manifest, and the completion publish still happen before the curated replace. A streaming job that dedupes in a store and then publishes the tile is a different product with a different state budget. The tile may count a duplicate event if the producer duplicated it. The morning job collapses event_id. If the live tile must be duplicate-free, that is a new requirement with a state store keyed by event_id, not a silent change to the count. Write it as a change to the tile contract.

Accuracy against the outside world stays a reconciliation. Deduped (order_id, line_id) in the raw prefix versus the table. Cohort keys versus enrollment. Closed-window counts versus a quiet hour of course_day, with an allowed gap. Those comparisons can run every few minutes. They are still set checks. They do not fit inside a single message transform.

Freshness SLOs you can calculate

Define them in UTC, on event time or commit time, and name the measurement point.

Activity tile. A closed five-minute window is visible within two minutes after stream time or watermark time passes the window end plus the ten-minute grace. The ten minutes are lateness, not the SLO. The SLO is how long after closure the sink shows the row. Measuring now minus the maximum event_time on the raw topic and calling that the tile SLO double-counts grace and ignores a stuck query that has not committed. Measure the query's last progress and the latest window actually written. A query that is up and a sink that is empty can be the open-window situation from lesson 45, which is healthy until a later event should have closed the window and did not.

Cohort. A committed enrollment change is visible in current state within fifteen minutes of source.ts_ms. Measure now minus the maximum source.ts_ms the fold has applied. Page separately if the slot is inactive. A cohort that is four minutes stale with an inactive slot is a disk page first and a freshness page second. A cohort that is twenty minutes stale with an active slot and a growing consumer lag is a fold-consumer page. The formulas differ, so the pages differ.

Purchase raw prefix. A committed line is in the prefix within fifteen minutes of event_time, which is loose enough for the morning job's same-day need, plus a completeness promise: every line_seq the source has committed is either in the prefix or on the dead-letter topic. Freshness without completeness is the dead-letter hole. Completeness compares max(line_seq) at the source offset with distinct line_seq values in the prefix and the dead-letter topic, allowing for files the sink has not rotated yet. Do not promise completeness from object listings alone.

Revenue topic. updated_event_time on commerce.course_revenue moves within about a minute of the line's availability on commerce.purchase_line, plus the JDBC poll interval, which is the larger term. Page on the application.id group's record lag and on task error. Do not page because the commit interval is one second while the source only polls once a minute.

Write the SLO next to the timestamp field. A number without a field is a panel.

Schema failures and lateness

A producer incompatibility response from the registry is a failed deploy. Page the service that is deploying. Do not page the sink owner because the topic's end offset stalled. The sink is caught up to an end that stopped moving.

Dead-letter growth on the S3 sink is a quarantine. Page the sink owner when the rate is non-zero for a financial topic. For progress, page when the rate exceeds the quarantine cap lesson 17 already used for bad rows, so one corrupt client does not wake the same person as a total converter outage. Read headers before payloads.

Dropped late records on the tile are a warn unless the rate says grace no longer matches client behavior. The correction is the morning restatement, not a replay into a closed window. A spike in drops together with one future-dated event_time is a poison clock. That one is a reject, from lesson 45, and it is a different alert from steady lateness.

Registry availability is a dependency of new schema ids. Alert on registry error rate and on the schema topic's disk. A consumer that has cached its ids survives a short outage. A deploy does not.

The lakehouse handoff

Module 5 starts from a raw prefix, not from a consumer group. The handoff for each live feed is a short contract.

The lakehouse commit, an Iceberg snapshot or a Delta version, creates a second freshness clock: when readers can see a table version. That clock belongs to Module 5, starting at Data Lake Architecture and Data Lakehouse Architecture. Landing an object is necessary and is not the same event. A query engine pointed at the prefix, which lesson 42 refused as a product interface, will see a partial hour and report a freshness the table does not have. The streaming SLO stops at the prefix and the topic. The table SLO starts at the commit. Write both, or the sink's on-call will be paged for a compaction job they do not run.

Small files are a cost metric on the prefix: objects per hour versus flush.size. They are not a quality failure unless readers time out. They become lakehouse maintenance input. Mention them in the handoff so Module 5 does not meet a million objects as a surprise. Do not block the tile on a compaction policy.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph cursors [Cursors]
        ckpt[Spark checkpoint]
        sinkg[Connect sink group]
        jdbc[JDBC line_seq offset]
        slot[Postgres slot WAL]
        fold[Cohort consumer group]
        streams[Streams application.id]
    end

    subgraph signals [Signals that page]
        stopped[Task FAILED or query down]
        dlq[Dead-letter rate]
        wal[Retained WAL while inactive]
        fresh[Event-time freshness past SLO]
        schema[Registry rejection on deploy]
    end

    subgraph notpage [Not a page by themselves]
        grace[Late drops inside grace budget]
        night[Overnight volume dip]
        zero[Empty result where a group does not exist]
    end

    ckpt --> fresh
    sinkg --> fresh
    sinkg --> dlq
    jdbc --> stopped
    slot --> wal
    fold --> fresh
    streams --> stopped
    schema --> stopped

Each page names one cursor. The box that is not a page is where enrollment week and the tile's grace belong, so they stay out of the fail path.

mermaidCopy
sequenceDiagram
    participant Prod as Producer or WAL
    participant Log as Topic
    participant Sink as Landing or fold
    participant DLQ as Dead letter
    participant Lake as Module 5 commit

    Prod->>Log: Record with event time
    alt Record is readable
        Sink->>Sink: Advance the real cursor
        Sink->>Lake: Object or row the table job can commit
    else Record fails the sink
        Sink->>DLQ: Quarantine with error headers
        Note over Lake: Completeness counts DLQ plus landing
    end
    Note over Lake: Table snapshot time is a second clock

The lake participant is a handoff, not a system this lesson deploys. Completeness includes the dead-letter topic so a green landing job cannot hide skips.

Technical Explanation

Commands that are the measurement

Sink lag, after you have confirmed the group id:

bashCopy
kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 \
  --describe --group connect-purchase-line-s3

Read LAG per partition. One stuck partition is a skewed key or a poison record the handler did not fail. Even lag across partitions is a slow sink or a paused task. The same command against the Streams application.id is the revenue job. If LOG-END-OFFSET is moving and CURRENT-OFFSET is not, the task is stuck, not idle.

Connector status is the Connect REST resource for that connector name, under /connectors/NAME/status. connector.state and each task state are the fields. FAILED is a stop. RUNNING with lag is slowness. RUNNING with dead-letter growth is a quarantine. The trace on the failed task is the first thing you read. Restarting in a loop without reading the trace repeats the same failure into the status topic.

Slot retention is the SQL from lesson 46. Put active and retained bytes on the same panel as the cohort consumer lag so the three cursors are visible together. A panel that shows only cohort freshness hides the disk risk until PostgreSQL stops accepting writes.

The Spark streaming query exposes input rate, batch duration, and the last watermark in its UI. Use those for the tile. Do not scrape a consumer group to infer them. If you export metrics, export the query's committed offset and the topic end offset, and name the series so it cannot be graphed as a Kafka consumer's records-lag. The name is yours. The source offsets refer to a topic. The commit is Spark's.

A small evaluator

The page decision can be a function over numbers you already collected. It does not call Kafka.

pythonCopy
from dataclasses import dataclass


@dataclass(frozen=True)
class Freshness:
    age_seconds: float
    slo_seconds: float

    @property
    def breached(self) -> bool:
        return self.age_seconds > self.slo_seconds


@dataclass(frozen=True)
class SlotHealth:
    active: bool
    retained_bytes: int
    disk_page_bytes: int

    @property
    def pages_disk(self) -> bool:
        inactive = not self.active
        over_budget = self.retained_bytes >= self.disk_page_bytes
        return inactive or over_budget


def purchase_completeness(max_line_seq: int, landed_or_dlq: int) -> int:
    """Identity values that are neither landed nor quarantined.

    A negative result means duplicates or a bad max. Treat it as a
    measurement bug, not as extra revenue.
    """
    return max_line_seq - landed_or_dlq

Call Freshness with the age you computed from the field the SLO names. Call SlotHealth with the query result. Feed purchase_completeness a landed_or_dlq count that includes the dead-letter topic. If you omit the dead letters, the function reports a hole you already quarantined, and someone will replay those records into a second copy.

The collector that fills the arguments is the exporter or the cron that runs the shell and the SQL. Keep the threshold numbers in the same repository as the contract. A threshold edited on a dashboard at 01:00 will not match the review.

Enrollment week and volume

Lesson 17 refused to fail a batch because enrollment week is larger than last Tuesday. The same rule applies to record rate. A floor of "at least the median of the last seven days" will page every term start. Use a floor only where silence is impossible, such as zero purchase lines for six hours while the storefront is returning success for checkouts, and pair it with that storefront signal. Do not page on progress-event rate alone during a launch. Do page on zero progress events while a live class roster says the session is running, if that roster is a real input. A volume warn without a second signal is noise.

Security of the monitor

Lag commands and connector status reveal topic names and sometimes exception text that includes a row. The dead-letter topic contains the failed payload. For progress, that can include a learner id. Restrict read access on dead-letter topics the way you restrict raw. Do not ship dead-letter bodies to a general logging vendor to make the alert pretty. Ship the rate, the connector name, the exception class, and the offset. A human who is allowed to see raw reads the body when the rate is non-zero.

The replication role's password and the Connect status port are production credentials. An unauthenticated status port on an open network is a configuration dump. Bind it the way the Connect security documentation describes for your deployment, or put it behind the same controls as the brokers. This lesson does not invent an authentication plugin. It refuses an anonymous status endpoint on a network learners can reach.

Practical Implementation or Design Walkthrough

Problem

A monitoring plan for three products: the progress tile and its raw landing, the purchase prefix, and the enrollment cohort. Each plan has a cursor, an SLO, a quarantine, and a page that is not the other product's page.

Progress

1. The tile query is up, the last watermark advances when events exist, and checkpoint lag is committed offsets versus end offsets. Page if the query is down, or if a window that should be closed has not landed within two minutes. A window should be closed when an event exists past the window end plus the ten-minute grace.
2. Late-drop rate is a warn. Future-dated reject rate is a page to the producer owner.
3. The S3 sink group lag and the dead-letter rate cover the landing the morning job reads. Completeness of the prefix is the morning gate's job to finish. The dead-letter rate is the live signal that the gate is about to see a hole.
4. Do not add Spark checkpoint lag and Connect sink lag into one number labeled progress lag.

Purchases

1. The source task is RUNNING, and the line_seq offset moves when max(line_seq) moves. A stalled offset with a growing table is a page to the connector owner.
2. Sink group lag stays inside the fifteen-minute freshness budget at the current produce rate. Better: measure now minus the maximum event_time in the newest object, so a quiet hour does not look like lag.
3. The dead-letter count is inside the completeness function. Any non-zero dead-letter rate pages during business hours, because a skipped line is revenue evidence. Replay is a tracked incident, not an automatic retry into the same bad converter.
4. A scheduled SQL check, which may live with the batch scheduler, compares distinct (order_id, line_id) in the prefix with the table for line_seq ranges the sink has closed. Duplicates are allowed in raw. Missing keys are not.

Enrollment

1. Slot active and retained bytes page at a disk threshold, and they page when active is false for more than a few minutes during write traffic.
2. Cohort consumer lag, and freshness as now minus the maximum applied source.ts_ms, against fifteen minutes.
3. A daily assertion that the completion mart's row count does not drop when op=d spikes. That assertion guards the wrong subscriber. It can be a batch check. It belongs on this plan because the failure mode is the streaming topic.

One board

Four rows, not forty charts: progress tile freshness, progress landing dead-letter rate, purchase completeness gap, and enrollment slot retained bytes next to cohort freshness. Schema-registry rejections sit on the deploy board. Streams revenue lag sits with commerce, beside purchase completeness, because a perfect prefix and a stuck aggregate means the intra-day total is stale while the books are safe.

Review the board by causing each failure in a lab once. Stop the Debezium task. Push one bad record through the purchase sink. Stop the Spark query. Send a future-dated progress event. If the board stays green, the board is decorative. Record the result in the runbook. Lesson 23's pipeline tests cover the batch gate. This lab is the streaming equivalent: one fired signal per failure mode.

Enterprise Real-World Scenarios

The green sink and the short hour

During a promotion, purchase-line-s3 stays RUNNING and object counts rise. A new field the Parquet writer cannot accept starts failing conversion. errors.tolerance=all sends those records to commerce.purchase_line.dlq and commits the sink offset. Record lag stays low. Event-time freshness of the objects that did land stays inside fifteen minutes. Completeness, max(line_seq) minus landed and dead-lettered identities, is zero only when the dead-letter topic is included. A dashboard that ignores it reports a hole of several thousand lines. Someone who sums the prefix as live revenue understates the promotion. The page goes to the sink owner with the exception class from the header, not to the database owner. The source task is healthy. The morning job quarantines nothing, because the bad records never arrived, unless the completeness check fails the publish. Wire the dead-letter count into that check or the batch gate cannot see the skip. The dead-letter payload is still a purchase line. The alert carries the offset and the error class. The on-call who is allowed to see raw reads the record. Freshness was the wrong SLO to stare at. The close stays honest only if completeness can fail the curated publish.

The cohort looks fine until the primary does not

A connector deploy leaves enrollment-debezium in FAILED at 10:00. Cohort freshness is four minutes, inside fifteen, because the last events were recent and no new unenrollment has happened. A board that shows only freshness stays green. Retained WAL grows with every other write on the primary, including progress writes the slot does not capture, because the slot cannot confirm. At 10:20 the disk threshold pages the database owner and the connector owner together. They start the task after fixing the config. WAL retention falls. The cohort never missed a delete. The incident was disk. The freshness SLO was the wrong severity. The follow-up is the slot panel at the top of the streaming board, and a deploy rule that a failed Debezium task is rolled back before the deployer leaves, not when the cohort looks stale. The fifteen-minute product SLO did not represent the risk. The slot metric did.

Best Practices

• Name the cursor in the alert. "Lag" without a group id or a checkpoint path is not an alert.
• Compute event-time freshness from the field in the contract. Keep record lag beside it as a diagnostic, not as a synonym.
• Do not look for the Spark tile in consumer-group lag.
• Include dead-letter counts in completeness. A skipped financial record is a failed check, not a successful sink commit.
• Page when the enrollment slot is inactive. Freshness can look fine while disk fills.
• Keep late-event drops as a warn unless they exceed the grace budget you set with the product.
• Send registry rejections to the deploying service. Send poison clocks to the progress producer. Send sink dead letters to the landing owner.
• Leave uniqueness, referential checks, and the curated publish on the lesson 17 gate.
• Hand Module 5 a prefix, a time field, a quarantine topic, and an explicit statement that table-commit freshness is not topic lag.
• Count objects per hour so small files are visible before they are a lakehouse incident.
• Restrict dead-letter reads. Alert on rates and exception classes.
• Fire each page once in a lab and paste the result into the runbook.
• During enrollment week, do not tighten volume floors. Keep completeness and slot health, which do not depend on last week's median.
• Authenticate the Connect REST listener. Status traces are operational data.

Common Mistakes and Anti-Patterns

One lag number for every pipeline. The Spark checkpoint, the sink group, and the slot are different failures. An average pages nobody who can act, and it clears when any healthy consumer is busy.

Declaring the Spark job healthy because a group-lag query returned nothing. No group is not zero lag. Read the checkpoint.

Freshness measured as wall clock minus ingest time, and labeled event time. Offline clients look late, or they look on time, depending on the mistake, and the tile's grace is ignored. Use event_time or source.ts_ms as written in the contract.

Paging on every late record. Grace exists so those records drop. A page per drop trains on-call to ignore the poison-clock page that matters.

A dead-letter topic with no alert, or an alert with no owner. The sink stays RUNNING. The prefix is incomplete. Lesson 42 built the topic so this lesson could count it.

Restarting a failed Debezium task on a timer. The slot stays inactive between crashes, WAL grows, and the trace nobody read is a bad password. Restart after the trace has an owner.

Streaming a full dimension join so the live path can fail referential integrity. That is a new stateful job. The morning gate already pins the manifest. A warn flag, as on the revenue topic, is enough while the line is live.

Calling raw-prefix freshness the lakehouse SLO. The table commit can lag the object by a compaction interval. Analysts will quote whichever number is smaller. Publish both clocks, or only the one the reader actually uses.

A volume page tuned in a quiet week. Term start looks like an incident. Completeness and slot health do not have that false positive. Prefer them.

Putting payloads in the platform log to debug a dead letter. Learner ids and purchase lines do not belong in a shared log sink. Offsets and error classes do.

Troubleshooting and Design Review

Symptom: The progress tile looks stale. Consumer-group lag for every group you know is near zero.

Likely cause: you are not looking at the Spark checkpoint. The query is stopped, or the watermark has not closed a window, or the dashboard reads a different sink path.

Investigation: the streaming UI's last progress and last watermark. The checkpoint directory's modification time. Whether a later event exists that should have closed the window.

Corrective action: start the query from the existing checkpoint if it is down. If the window is legitimately open, do not page freshness yet. If the checkpoint was lost, follow lesson 32's replay-or-skip decision. Do not commit a consumer group to catch the query up.

Symptom: Purchase sink lag is zero and the morning completeness check fails.

Likely cause: dead-lettered records. The sink committed past them. Lag of the records it accepted is zero.

Investigation: the dead-letter end offset and the exception class on the error header. Compare line_seq values there with the table.

Corrective action: fix the converter or the schema. Replay the dead-letter topic through a corrected path, or re-read the original offsets if retention allows. Add the dead-letter count to the live completeness number so the next miss is not a morning surprise.

Symptom: Cohort freshness is inside the SLO and the primary disk alarm fires.

Likely cause: an inactive slot and retained WAL, with little new enrollment traffic, so the cohort looks current.

Investigation: pg_replication_slots for enrollment_debezium, and the connector task state.

Corrective action: restore the task. Do not drop the slot to clear disk unless the database owner accepts a rebuild. Split the alert so disk and freshness are both visible, and so disk is not gated on freshness.

Symptom: Late-drop warnings fired all morning, then a future-dated event appears on the reject topic.

Likely cause: the future-dated event moved stream time, and ordinary events became late. The warn was the symptom. The reject is the cause.

Investigation: the maximum event_time per partition in the last hour, compared with broker time.

Corrective action: fix the producer clock. Rebuild the tile's window state for the affected hour if you served wrong counts, from the source topic, with a new checkpoint or a new application.id. Do not widen grace to swallow the poison hour.

Symptom: The lakehouse table is hours behind and every streaming SLO is green.

Likely cause: the table commit job is behind. Objects are landing. The second clock is stopped.

Investigation: snapshot or version time on the table, versus the newest object time in the prefix.

Corrective action: page the table job's owner. Do not retune sink flush.size as the first response. The handoff contract should have made this clock explicit so the page is not a debate.

A design review asks: for each product, what is the cursor, the timestamp field, and the SLO? What is quarantined rather than skipped? What stops the task? What only warns? Where is the slot panel? How does a dead letter affect completeness? Which number is the lakehouse not allowed to quote as its own freshness? Who can read the dead-letter topic? Has each page been fired once?

Interview Questions and Answers

Q1. What is the difference between record lag and event-time freshness?

Record lag is how many records sit between a consumer's committed offset and the log end. It is the LAG column from kafka-consumer-groups.sh. Event-time freshness is how old the newest applied event_time, or the newest source commit time, is compared with now. A consumer can be caught up in records while the newest event is old, because producers were quiet or clients were offline. A consumer can be far behind in records while the prefix it has applied is still inside the SLO, with the unread tail about to break it. I put both on the board and I page on the one the contract names. A low record lag is not proof the tile is current.

Q2. Why is consumer-group lag the wrong health check for the Spark activity tile?

The tile's cursor is the checkpoint from lesson 32. Structured Streaming does not commit that cursor to a Kafka group. A lag query that returns nothing, or that returns some other group, looks like health while the query is stopped. I read the query's committed offsets against the topic end, and I use the streaming UI for the watermark. If I export a metric, I name it as checkpoint lag so it is not graphed as a consumer.

Q3. The purchase sink is RUNNING and lag is zero, but lines are missing. Where do you look?

The dead-letter topic. With errors.tolerance=all, the sink commits past records it could not write and stays running. Lag measures records it has not read yet, not records it quarantined. I count identities in the landing plus the dead-letter topic against max(line_seq). A gap is a page. A non-zero dead-letter rate on this financial sink is already a page during the business day. Freshness of the objects that landed can still meet the SLO, which is why freshness alone is the wrong board.

Q4. How should the enrollment slot appear next to cohort freshness?

As its own signal. An inactive slot, or retained WAL above a disk threshold, pages even when now minus the latest applied source.ts_ms is inside fifteen minutes. A quiet enrollment table makes freshness look healthy while the slot holds WAL from every other write on the primary. I do not gate the disk page on the freshness formula. I also show the cohort consumer's record lag, because an active slot and a stuck fold is the freshness incident. Three cursors, three numbers.

Q5. Which streaming defects fail the task, which are quarantined, and which only warn?

I fail the task when moving the cursor would skip evidence: Debezium errors, JDBC source errors, a purchase Streams deserializer failure, and Spark offset loss. I quarantine when the record can sit aside and the rest of the stream is still valid: sink dead letters, and progress events rejected for a missing or future event_time when that path is a reject topic. I warn on late drops inside the grace budget and on volume dips that enrollment week would mimic. The morning curated gate still fails the publish for uniqueness, referential breaks, and completeness. I do not reimplement that gate inside a consumer and then skip the batch one.

Q6. How do you write the tile's freshness SLO so grace is not counted as an outage?

The product already waits ten minutes of watermark or grace before a five-minute window closes. The SLO I page on is the delay after that closure: the closed window is visible within two minutes. I do not page because now minus event_time is twelve minutes on a window that is still open. I do page when a record newer than the window end plus grace has been processed and the closed count is still absent. Late drops stay a warn. That matches lessons 32 and 45, where an unclosed window means the system is waiting for a later timestamp.

Q7. A registry rejection stalls the purchase topic. Who do you page, and what do you not restart?

I page the producer deploy. The registry refused a schema, so the client stopped sending, and the sink is correctly caught up to a quiet end. Restarting the sink, the JDBC source, or the Streams aggregate does not register a compatible schema. I also do not lower compatibility to NONE to clear the page. The topic is quiet because the contract held.

Q8. What do you hand Module 5, and which number must the lakehouse not quote as table freshness?

I hand the raw prefix, the UTC path, the event-time field, the dead-letter topic, and the completeness rule that counts quarantined records. I hand the cohort topic and the fold's table as current state, not as a substitute for the daily snapshot. The number the lakehouse must not quote is topic lag or the time of the newest object listing. A table is fresh when its snapshot or version commits. That second clock belongs to the lakehouse job. If both clocks are not written down, the sink on-call carries compaction lag.

Q9. Why is a volume drop a bad page during enrollment week, and what do you page on instead?

Last week's median rate is not a property of a launch week. Lesson 17 made the same point for batch volume checks. I page on completeness gaps, dead letters, stopped tasks, inactive slots, and freshness formulas tied to a timestamp. I would page on zero purchase lines only together with a second signal, such as the storefront serving checkouts. Rate without that signal is a warn with a review date, or it is not an alert.

Q10. How do you keep a monitor from leaking the records it protects?

Dead-letter topics and connector traces can contain learner identifiers and purchase lines. Alerts carry the connector name, the partition, the offset, and the exception class. People who already have raw access read the payload. I do not attach the record to a chat webhook. The Connect status endpoint is not anonymous on a network users can reach. Lag metrics are not the sensitive part. The quarantine store is.

Q11. Spark checkpoint lag is high and the Connect progress sink lag is high. What do you restart?

Only the cursor that is late, after you know why. The tile and the landing are two readers of learning.progress. Restarting a healthy Spark query because the sink is behind risks a checkpoint mistake and does not write the objects. Restarting the sink because the tile is behind does not advance the watermark. I page the owner of the late cursor. If both are late because the brokers are not serving the topic, I page the cluster, not both application teams as if they had independent bugs.

Hands-On Exercises

Exercise: A board that can tell the failures apart

Problem statement. Write the monitoring plan for the progress landing, the purchase prefix, and the enrollment slot. Fire two lab failures and show that two different signals trip, without a false page from grace or from a quiet hour.

Tasks.

1. For each product, write the cursor, the timestamp field, the SLO, the quarantine location, and the owner. One table is enough.
2. Implement Freshness, SlotHealth, and purchase_completeness. Unit-test a breach, an inactive slot, and a completeness gap that disappears once the dead-letter count is included.
3. In a lab, stop the Debezium task and record slot active and retained WAL. Show that freshness computed from the last applied event can still sit inside fifteen minutes.
4. Send one bad purchase record through a sink with a dead-letter topic. Show sink lag returning to zero while completeness, counted without the dead-letter topic, is short.
5. Using the lesson 32 and 45 contracts, describe a progress window that is open and not an incident, and one that should have closed and is an incident.
6. Write the Module 5 handoff: what the table job may call freshness, and what it must not.
7. State one alert you will not create for enrollment week, and the completeness or slot alert you will keep.

Expected deliverables. The plan table, the unit tests, the lab notes for the slot and the dead letter, and the handoff paragraph.

Evaluation rubric.

You pass when an inactive slot is a page while freshness is still inside the SLO, and when a dead-lettered purchase line shows up in the completeness gap and does not show up in sink lag.

Lesson Summary

Streaming quality is a set of cursors and severities. It is not a second morning gate and not a single lag chart. Record lag counts unread records for a real consumer group. Event-time freshness subtracts the contract's timestamp from now. The Spark tile's cursor is its checkpoint. The purchase sink can be caught up and still incomplete when the dead-letter topic holds the gap. The enrollment slot's retained WAL is a disk page even when the cohort looks fresh. Late drops inside grace warn. Registry rejections page the deployer. Uniqueness and referential checks stay on the batch publish. Module 5 receives prefixes, quarantine topics, and a separate clock for the table commit.

Data Lake Architecture is where those prefixes become a layout with owners. The live path you just measured is the input, not the table.

Further Reading

• Kafka operations and monitoring — broker and client metrics. The consumer-group LAG column is the record-lag measurement in this lesson.
• Connect monitoring — task metrics and the connector status resource.
• Basic consumer-group operations — describing a group, and why a missing group is not a zero.
• PostgreSQL logical decoding — WAL retained while a slot is inactive.
• Data Quality Engineering — severities and the curated gate streaming does not replace.
• Data Lakehouse Architecture — the commit clock this lesson does not pretend to measure.
• Data Observability and Operational Monitoring — estate-wide monitoring after this streaming board exists.
• /learn/kafka — the log end and the committed offset that record lag subtracts.