Data Observability and Operational Monitoring

Data Engineering & Modern Data Platforms · Lesson 60 · Production Data Engineering and Capstone

Lesson Overview

The warehouse is up. The DAG is green. Forty people are looking at a board that stopped moving yesterday. Freshness failed quietly because the only alert you had was "job failed," and the job did not fail. It published zero new attempts for a day that should have held a weekday's progress. Volume collapsed. Nobody paged. The status channel stayed calm until a human opened a chart and asked why Monday looked like Sunday.

DataOps and Data Platform Engineering named owners and change gates. This lesson gives those owners eyes. Data observability is the practice of monitoring the data itself—freshness, volume, schema, nulls, distribution, lineage impact—not only the CPU of the cluster that wrote it. Operational monitoring still matters: DAG duration, consumer lag, warehouse queue depth, error rates. The two layers answer different questions. Green infrastructure with wrong data is how TechLearningPro loses trust. Red infrastructure with healthy data is how you burn the on-call without a board defect.

The teaching story stays fixed. About 40 GB of compacted Parquet progress a day. A completion mart of tens of gigabytes. Publish with margin before 06:00. definition_version pinned. Lesson 47 separated streaming quality cursors from certified grain. Lesson 17 taught quality expectations. Lesson 22 taught lineage. This lesson turns those ideas into SLIs, alert routes, and a paging policy that does not wake people for every null blip on an optional column.

Vendors sell "data observability" suites. The concepts—anomaly detection on volume, schema drift alerts, lineage-based blast radius—are portable. This lesson uses those concepts without locking you to a product API. You can implement monitors with SQL, warehouse INFORMATION_SCHEMA, open-source quality libraries, metrics backends, and sober thresholds. Data Reliability and SLA/SLO Design will formalize error budgets. Arrive there knowing which signals are paging and which are tickets.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate infrastructure monitoring from data observability and say when each is insufficient alone.
2. Define freshness, volume, schema, null-rate, and distribution monitors for progress and the completion mart.
3. Choose SLIs that map to the Monday 06:00 board promise without drowning on-call in noise.
4. Use lineage to scope blast radius when an upstream schema or volume shock hits.
5. Decide paging versus ticket versus dashboard-only for each class of signal.
6. Detect silent success: jobs that succeed while publishing empty, duplicate, or wrong-version data.
7. Build a minimal observability loop in SQL and metrics without a proprietary vendor API.
8. Tie alerts to lesson 59 owners so the right rotation wakes up.
9. Explain Monte Carlo-style automated monitoring ideas as patterns: baselines, anomalies, incident clustering.
10. Hand lesson 61 monitors that do not require scanning raw PII to operate, and hand lesson 62 a candidate SLO signal list.

Prerequisites

• Lesson 59 for ownership, runbooks, and CI gates; alerts without owners are noise.
• Lesson 58 for dry-run bytes and isolation; a scan spike is an observability signal and a cost signal.
• Data Quality Engineering for expectation shapes.
• Real-Time Data Quality and Monitoring for streaming cursors versus certified grain.
• Data Lineage and Metadata Management for blast-radius queries.
• Apache Airflow Architecture and DAGs for job-level metrics.
• Kafka consumer lag lessons for the progress topic path when landing stalls.
• Lesson 62's upcoming SLO language so you do not invent three conflicting freshness definitions.

Detailed Core Concepts

Two planes of truth

Operational monitoring watches systems: Is the orchestrator scheduling? Is the warehouse accepting connections? Is consumer lag growing? Are error rates up? Is disk full?

Data observability watches datasets: Did today's partition arrive by the deadline? Is row count inside a seasonal band? Did columns appear, disappear, or change type? Did null rates jump? Did a key that should be unique stop being unique? Did downstream tables that depend on this one go stale in a chain?

You need both. A Kafka cluster can be healthy while producers stopped sending progress. A Spark job can succeed after writing an empty overwrite because an upstream filter was wrong. A warehouse can be fast while serving completion-v3 after product cut over the board SQL to v4 expectations—or the reverse.

The monitor types that earn their keep

For TechLearningPro's certified path, standardize on these monitor classes:

Freshness. Time since last successful publish for event_date = today (or the board date). Compare against the SLO-shaped deadline: mart ready with margin before 06:00. Freshness is not "DAG finished"; it is "data consumers can read is new enough."

Volume. Row counts or bytes per partition versus a baseline. Weekdays and weekends differ. Course launches spike. A flat zero on a Tuesday is a defect. A twenty percent drop on a holiday may be normal. Baselines need calendar awareness.

Schema. Column set, types, and nullability versus the contract. Alert on unexpected columns in certified tables (may indicate star-select landings) and on missing required columns. Schema monitors catch what CI missed when someone changed production outside the gate—lesson 59's failure mode.

Null and validity rates. Fraction null for required fields (lesson_id, attempt_id). Fraction failing format checks. A slow creep of nulls is as dangerous as a cliff; both need thresholds.

Uniqueness and grain. Duplicate attempt_id rates. Uniqueness is a correctness SLI, not a vanity metric.

Distribution / anomaly. Relative changes in categorical mixes (share of mobile vs desktop clients, share of a lesson_id) when you have a reason to care. Unsupervised anomaly detection helps at scale; it also false-positives. Start with a few supervised metrics tied to product meaning.

Lineage freshness fan-out. When progress landing is stale, mark compacted Parquet and the mart as suspect even if their jobs have not run yet. When the mart is stale, mark the board extract as suspect. Lineage turns one root cause into a targeted incident instead of forty confused readers.

Job and queue health. DAG duration, task retries, warehouse queue time, slot contention, consumer lag. These explain freshness failures and sometimes predict them.

SLIs, not metric hoarding

A service level indicator (SLI) is a carefully chosen measurement of user-visible health. For the completion board, strong SLIs include:

1. Fraction of Mondays where published.completion_attempt for the board date range is queryable with the pinned definition_version before 06:00.
2. Freshness lag minutes for the latest event_date partition relative to the publish deadline.
3. Rate of certified queries that fail grain checks in the last twenty-four hours.
4. Percent of board refreshes served from the mart versus accidental raw scans (ties to lesson 58).

Weak "SLIs": CPU average, number of alerts fired, count of tables with any monitor. Those can be operational metrics. They are not board health.

Lesson 62 will bind SLIs to SLOs and error budgets. Your job here is to instrument the candidates and keep noise low enough that people still trust pages.

Paging versus tickets versus dashboards

If everything pages, nothing pages. Lesson 59 ownership decides who. This lesson decides whether they wake up.

Silent success and the empty publish

The most important observability lesson for batch systems: success is not freshness and success is not correctness. Emit and alert on:

• Rows written (or files committed) per partition.
• Whether the Iceberg/Delta snapshot advanced for the expected partition.
• Whether definition_version in the new data matches the expected current version.
• Whether the board date exists as a partition at all.

A job can exit zero after an upstream WHERE 1=0. Your freshness monitor should see "no new data" even when Airflow is green. Wire job success and data success as different signals. Correlate them in the runbook: green job + stale data = transform logic or upstream empty; red job + fresh data = you might be looking at yesterday's successful publish still serving.

Baselines and seasonality

Simple thresholds (rows > 1000) fail on weekends and launches. Prefer:

• Compare to the same weekday over the last N weeks.
• Allow absolute floors for "never zero on a teaching weekday."
• Maintain a calendar of known low days (company holidays, platform maintenance).
• For launches, expect upper anomalies; alert on unexpected silence, not only on spikes.

Vendor products automate baselines. You can approximate with SQL window functions and a metrics table you own. Automation without an owner still pages the wrong person.

Lineage alerts without theater

You do not need a perfect graph of every spreadsheet. You need edges for certified assets: progress topic → landing → compacted Parquet → completion mart → board SQL / extract. When a monitor fires on landing, open an incident that lists downstream certified consumers from that graph. When someone proposes dropping a column, the same graph lists who breaks. Lesson 22's catalog work pays off here only if edges are maintained for the certified path. Exhaustive lineage of every ad-hoc query is a project that never finishes; start with the Monday board chain.

Monte Carlo-style concepts without the logo

Commercial data observability popularized a bundle of ideas worth separating from the SKU:

1. Always-on monitors across many tables, not only the ones someone remembered to test in CI.
2. Anomaly detection on volume and freshness instead of only static thresholds.
3. Incident grouping so one upstream break does not create forty pages.
4. Lineage-informed prioritization of which break threatens a CEO dashboard.

Implement those ideas with: a monitor registry (YAML), scheduled SQL checks, a metrics backend, a grouping key (root_asset_id), and ownership from lesson 59. Buy a platform later if the registry becomes the pain; do not skip the registry and hope a vendor invents your ownership model.

Privacy-aware observability

Monitors should prefer counts, hashes, and rates over sample rows that contain email. Lesson 61 will harden this. Design now so debug "show me bad rows" lands in a restricted tool with access logs, not in a Slack channel. Volume and null monitors do not need PII. Distribution monitors on lesson_id are fine; distribution monitors on raw email domains may not be.

What "Monte Carlo-style" means in engineering language

Teams often say they want "Monte Carlo" when they mean four capabilities: automated profiling of many tables, anomaly detection on freshness and volume, a UI that clusters related failures, and lineage that answers who is downstream. None of those require a single vendor. Profiling is scheduled COUNT, COUNT DISTINCT, and schema snapshots. Anomaly detection starts as seasonal baselines and can graduate to robust statistics or simple forecasting. Clustering is a root_asset_id and a silence window. Lineage is OpenLineage events or a curated YAML graph for certified assets. Buy software when maintaining those four is harder than paying; do not buy software instead of naming owners and SLIs.

Synthetic checks versus data checks

A synthetic check runs the certified board SQL on a schedule with a service identity and records latency and row counts. It proves the read path works, including permissions and warehouse availability. A data check proves the underlying partition is fresh and well-formed. Use both for the Monday board. Synthetic alone can pass on yesterday's data if you forget the date filter. Data alone can look healthy while the BI tool's identity cannot read the table. Lesson 57's identities show up here as a monitoring concern, not only a security concern.

Error budgets preview (without stealing lesson 62)

If the board must be ready before 06:00 on every Monday, a single miss is already painful. Still, track misses: minutes late, empty publishes, duplicate incidents. That history is the input to an SLO. Observability that deletes history after the page resolves leaves reliability blind. Store monitor outcomes in a durable table: monitor_id, asset, status, value, routed_as, incident_id. Lesson 62 will compute burn rates. You should already be recording the raw events.

Cross-cloud and multi-engine honesty

If a reopen condition from lessons 54–56 ever dual-publishes, monitors must cover both engines or you must declare one non-certified. A green freshness monitor on the primary warehouse while the abandoned secondary drifts is how shadow consumers get wrong numbers. DataOps forbids quiet dual publish; observability detects it if someone does it anyway by watching unexpected writers or second-engine row counts for the same grain.

Alert fatigue physics

Every page trains the human. If one in three pages is a false positive, people slow-walk the response. Measure precision for paging monitors weekly for the first month: true user impact versus noise. Kill or demote monitors that cannot reach high precision. It is better to have five trusted pages than fifty ignored ones. Ticket routes can be noisier; pages cannot.

Architecture and Design Diagrams

Signals flow from data and jobs into a decision about who wakes up.

mermaidCopy
flowchart LR
  JOB["DAG and consumer metrics"] --> OPS["Operational monitors"]
  DATA["Mart and landing checks"] --> DOBS["Data monitors"]
  DOBS --> SLI["SLI evaluation"]
  OPS --> SLI
  LIN["Lineage graph"] --> GROUP["Incident grouping"]
  SLI --> GROUP
  GROUP --> PAGE["Page owner"]
  GROUP --> TICKET["Ticket"]
  GROUP --> DASH["Dashboard only"]

The certified chain is the minimum lineage you must watch.

mermaidCopy
flowchart LR
  T["Progress topic"] --> L["Landing raw"]
  L --> C["Compacted Parquet"]
  C --> M["completion_attempt mart"]
  M --> B["Monday board SQL"]
  M --> X["BI extract"]
  L -.->|freshness fail| M
  M -.->|freshness fail| B

Technical Explanation

Freshness as a cursor, not a vibe

Define freshness lag as:

lag_minutes = now() - max(publish_time) FILTER (WHERE event_date = :board_date AND definition_version = :pinned)

Alert when lag_minutes exceeds a paging threshold that still leaves response time before 06:00 on Mondays, and a tighter ticket threshold on weekdays if product wants same-day boards. Store the pinned version next to the monitor so a cutover does not false-page when v3 stops moving and v4 is healthy.

For streaming landing, a second cursor from lesson 47 may track provisional tiles. Do not mix cursors. The certified mart monitor never reads the provisional tile's success as mart freshness.

Volume checks that survive Monday

Pseudo-logic:

1. Compute rows for yesterday's event_date (or hour, if you truly partition by hour—lesson 58 argued you usually should not for this board).
2. Compute median_rows for the same weekday over the last six weeks, excluding known holidays.
3. Page if rows == 0 on a non-holiday weekday.
4. Ticket if rows < 0.5 * median_rows or rows > 3 * median_rows without a launch flag.
5. Launch flags are calendar entries owned by product, not guesses by on-call at 02:00.

Bytes can substitute when row counts are expensive; for the mart, row counts are cheap and clearer.

Schema diff mechanics

Nightly (or per-publish) compare INFORMATION_SCHEMA or catalog metadata to the contract file in git. Emit:

• missing_required_columns
• unexpected_columns
• type_mismatches

Unexpected columns may be allowed in raw landing and forbidden in the certified mart. Encode that in the monitor config. Schema alerts should page for certified mart breaks and ticket for raw-zone surprises unless raw feeds a tightly coupled streaming job.

Null rates and the slow bleed

Alert on step changes and on linear creeps. A column that goes from 0.1% null to 0.3% per week will miss a static threshold for a month while quality dies. Record a baseline null rate at cutover of v4 and ticket on relative growth. Page only if a required column's null rate crosses a hard ceiling (for example 1% of attempts missing lesson_id).

Duplicate grain detection

sqlCopy
SELECT COUNT(*) AS dup_groups
FROM (
  SELECT attempt_id
  FROM published.completion_attempt
  WHERE event_date = DATE '2026-11-10'
    AND definition_version = 'completion-v4'
  GROUP BY attempt_id
  HAVING COUNT(*) > 1
) d;

dup_groups > 0 on a certified day is a page for domain. Duplicates destroy rates. This is not a soft anomaly; it is a grain break.

Correlating job metrics with data metrics

Export Airflow task duration and state. Export warehouse queue time. When freshness fails:

1. If the DAG failed, fix the job; data monitors confirm recovery.
2. If the DAG succeeded and volume is zero, inspect filters and upstream volume.
3. If the DAG succeeded, volume looks normal, but board disagrees, inspect definition_version and extracts.
4. If queue time exploded and duration blew past the window, you have a capacity or isolation incident (lesson 58), not a quality rule incident.

Alert grouping and root assets

Give every monitor a root_asset_id. Landing volume and landing freshness share progress_landing. When both fire, one incident. When landing fires, automatically raise "suspected impact" on completion_mart without double-paging until the mart's own freshness threshold trips—or attach the mart owner as a stakeholder notification instead of a page. Tune this once; wrong grouping is either noise or missed wake-ups.

Sampling and cost of monitors

Observability jobs that full-scan raw JSON daily recreate lesson 58's bill. Run heavy checks on the mart and on partition metadata. Use table statistics, partition-level counts, and sampling for distribution checks. A monitor that costs more than the board it protects needs a redesign.

Partition metadata as a first signal

Before you COUNT(*) a huge table, ask whether the lake table's partition list or Iceberg/Delta snapshot summary already tells you the day is missing. A missing partition entry is a freshness failure with almost no scan cost. A present partition with zero rows is rarer but still cheap to detect with a partition-level stats table maintained at publish time. Have the publish job write ops.publish_stats(event_date, definition_version, row_count, byte_count, publish_time). Monitors read ops.publish_stats first. If the publish job lies, you have a deeper integrity problem; still keep a periodic independent recount on the mart for certified days.

Downstream extract monitors

BI extracts and materialized dashboard caches fail in their own way: they go stale while the mart is fresh, or they pin an old schema. Add a small monitor: extract refresh time, extract row count versus mart aggregate for the same filter, and extract column contract hash. When lesson 59 cuts over definition_version, extract monitors should fail until refreshed—that failure is desired. Treat it as a ticket during planned cutover windows and as a page if it happens unplanned on a Monday morning.

Runbook hooks from alerts

Every paging monitor links to a runbook section id. The page body includes: asset, monitor id, current value, threshold, owner, and runbook URL. Pages that say only "VolumeAnomalyCritical" waste the first five minutes. Lesson 59 wrote page one of the late-board runbook; observability should deep-link into those steps based on which monitor fired (freshness vs duplicates vs schema).

Multi-region and clock issues

If publish_time is stored in UTC and on-call thinks in local time, thresholds look wrong near midnight. Standardize on UTC in monitors and convert in the notification text. If a secondary region exists for disaster recovery (lesson 58), do not page on secondary freshness unless you have failed over. Watching a cold replica as if it were the board source creates permanent noise.

Practical Implementation or Design Walkthrough

Problem

Design a data observability dashboard and alert policy for TechLearningPro's completion path: progress landing through published.completion_attempt to the Monday board. Include freshness, volume, schema, null, and duplicate monitors; map each to page/ticket/dashboard; bind owners from lesson 59; avoid vendor lock-in.

Monitor registry

yamlCopy
# observability/monitors/completion.yaml
asset: published.completion_attempt
owners:
  page: learning-analytics-oncall
  ticket: learning-analytics-de
pinned_definition_version: completion-v3
monitors:
  - id: mart_freshness
    type: freshness
    sql_ref: sql/obs/mart_freshness.sql
    page_if_lag_minutes_gt: 45
    ticket_if_lag_minutes_gt: 20
    apply_hours: ["00:00-06:00"]  # tighten before board
  - id: mart_volume_weekday
    type: volume
    sql_ref: sql/obs/mart_volume.sql
    page_if_rows_eq: 0
    ticket_if_ratio_vs_weekday_median_lt: 0.5
  - id: mart_schema
    type: schema
    contract_ref: contracts/completion_attempt.yaml
    page_on: [missing_required, type_mismatch]
    ticket_on: [unexpected_column]
  - id: mart_null_lesson_id
    type: null_rate
    column: lesson_id
    page_if_null_rate_gt: 0.01
  - id: mart_dup_attempt_id
    type: uniqueness
    column: attempt_id
    page_if_dup_groups_gt: 0

Freshness SQL

sqlCopy
-- sql/obs/mart_freshness.sql
SELECT
  DATE_DIFF('minute', MAX(publish_time), CURRENT_TIMESTAMP) AS lag_minutes,
  MAX(definition_version) AS latest_version_seen,
  COUNT(*) AS rows_today
FROM published.completion_attempt
WHERE event_date = CURRENT_DATE
  AND definition_version = 'completion-v3';

Evaluator logic (separate job) reads lag_minutes, compares to thresholds, and checks latest_version_seen matches pin.

Volume SQL with weekday baseline

sqlCopy
-- sql/obs/mart_volume.sql
WITH daily AS (
  SELECT event_date, COUNT(*) AS rows
  FROM published.completion_attempt
  WHERE definition_version = 'completion-v3'
    AND event_date >= CURRENT_DATE - INTERVAL '56' DAY
  GROUP BY event_date
),
today AS (
  SELECT rows FROM daily WHERE event_date = CURRENT_DATE - INTERVAL '1' DAY
),
baseline AS (
  SELECT approx_percentile(rows, 0.5) AS median_rows
  FROM daily
  WHERE event_date < CURRENT_DATE - INTERVAL '1' DAY
    AND day_of_week(event_date) = day_of_week(CURRENT_DATE - INTERVAL '1' DAY)
)
SELECT
  (SELECT rows FROM today) AS rows_yesterday,
  (SELECT median_rows FROM baseline) AS weekday_median;

Engine-specific functions vary (approx_percentile, PERCENTILE_CONT, etc.). Keep the shape; adapt the dialect.

Schema check in Python (contract vs information schema)

pythonCopy
# tools/schema_guard.py
from __future__ import annotations

import yaml
from dataclasses import dataclass


@dataclass
class Diff:
    missing: set[str]
    unexpected: set[str]
    type_mismatches: list[tuple[str, str, str]]


def diff_schema(contract_path: str, actual: dict[str, str]) -> Diff:
    contract = yaml.safe_load(open(contract_path))
    required = {c["name"]: c["type"] for c in contract["columns"] if c.get("required", True)}
    actual_cols = set(actual)
    missing = set(required) - actual_cols
    unexpected = actual_cols - set(contract["columns"][i]["name"] for i in range(len(contract["columns"])))
    # simpler unexpected: anything not listed at all
    listed = {c["name"] for c in contract["columns"]}
    unexpected = actual_cols - listed
    mismatches = []
    for name, typ in required.items():
        if name in actual and actual[name] != typ:
            mismatches.append((name, typ, actual[name]))
    return Diff(missing=missing, unexpected=unexpected, type_mismatches=mismatches)

Wire actual from a warehouse information_schema query in the same job. Exit non-zero on paging conditions so the orchestrator can alert.

Operational metrics to export

• airflow_dag_last_success_timestamp{dag="completion_publish"}
• airflow_dag_duration_seconds{dag="completion_publish"}
• warehouse_queue_time_seconds{pool="board_interactive"}
• kafka_consumer_lag{group="progress_landing"}

Dashboard panels: freshness lag, yesterday rows vs median, DAG duration vs window, lag, duplicate groups. Alert rules live in the monitor registry, not only as screenshots.

Paging policy snippet

textCopy
Page learning-analytics-oncall when:
  - mart_freshness page threshold breached on a board morning
  - mart_dup_attempt_id > 0
  - mart_null_lesson_id > 1%
  - mart_schema missing_required or type_mismatch

Page data-platform-oncall when:
  - progress landing volume = 0 on weekday
  - kafka_consumer_lag page threshold breached AND landing freshness failing
  - warehouse board pool queue prevents queryability while mart data is fresh

Never page for:
  - optional column null ticketing thresholds
  - CPU alone
  - anomaly lesson-mix without product flag (dashboard)

What this walkthrough refuses

• Alerting on every table in the lake on day one.
• Slack'ing sample rows with emails.
• Full-scanning raw JSON for daily null rates.
• Paging platform for a wrong definition_version pin.

Publish stats writer (companion to the model)

pythonCopy
# transforms/completion/write_publish_stats.py
from datetime import date, datetime, timezone


def write_publish_stats(conn, *, event_date: date, definition_version: str, row_count: int, byte_count: int) -> None:
    """Emit a cheap monitor row in the same transaction boundary as publish when possible."""
    conn.execute(
        """
        INSERT INTO ops.publish_stats
          (event_date, definition_version, row_count, byte_count, publish_time)
        VALUES (%s, %s, %s, %s, %s)
        """,
        (
            event_date,
            definition_version,
            row_count,
            byte_count,
            datetime.now(timezone.utc),
        ),
    )

Monitors can read ops.publish_stats for near-zero-cost freshness and volume. Schedule a weekly independent COUNT(*) reconciliation for certified days to detect a lying publisher.

Evaluator pseudocode

pythonCopy
# tools/evaluate_monitors.py
from dataclasses import dataclass


@dataclass
class Outcome:
    monitor_id: str
    route: str  # page | ticket | dash
    value: float
    message: str


def evaluate_freshness(lag_minutes: float, page_gt: float, ticket_gt: float) -> Outcome:
    if lag_minutes > page_gt:
        return Outcome("mart_freshness", "page", lag_minutes, "freshness past page threshold")
    if lag_minutes > ticket_gt:
        return Outcome("mart_freshness", "ticket", lag_minutes, "freshness past ticket threshold")
    return Outcome("mart_freshness", "dash", lag_minutes, "ok")


def evaluate_volume(rows: int, median: float) -> Outcome:
    if rows == 0:
        return Outcome("mart_volume_weekday", "page", 0, "zero rows")
    if median > 0 and rows < 0.5 * median:
        return Outcome("mart_volume_weekday", "ticket", rows / median, "below half median")
    return Outcome("mart_volume_weekday", "dash", rows, "ok")

Persist outcomes; fire notifications through your existing pager (email, PagerDuty-compatible webhook, etc.) without inventing a proprietary observability API.

Dashboard query for the 05:30 screen

sqlCopy
SELECT
  m.monitor_id,
  m.status,
  m.value,
  m.routed_as,
  m.observed_at
FROM ops.monitor_outcomes m
WHERE m.observed_at >= CURRENT_TIMESTAMP - INTERVAL '24' HOUR
  AND m.asset = 'published.completion_attempt'
ORDER BY
  CASE m.routed_as WHEN 'page' THEN 0 WHEN 'ticket' THEN 1 ELSE 2 END,
  m.observed_at DESC;

Enterprise Real-World Scenarios

TechLearningPro: green DAG, empty Tuesday

A filter bug in the completion model references a new lesson taxonomy table that was empty in production. The DAG succeeds in two minutes—faster than usual. Airflow is green. The freshness monitor pages at 05:10 because rows_today = 0. Domain on-call follows lesson 59's runbook: publish cursor empty, version string unused because no rows. They revert the model, republish, and the board meets 06:00 with minutes to spare. Without the volume/freshness page, the empty publish would have been discovered in the meeting.

Northline: forty alerts, one root cause

Northline fans out order events into twelve marts. A producer outage stops orders. Twelve volume monitors page twelve owners. On-call chaos burns an hour. After regrouping, they set root_asset_id: orders_landing and stakeholder notifications for downstream marts. One page to ingestion; tickets to domains labeled "suspected impact." The Monday portal SLI still pages the portal owner if the portal mart itself goes stale past its threshold—because user impact deserves a page even when root cause is known.

Anomaly theater on a course launch

Product launches a new certification track. Attempt volume triples. Untuned anomaly detection pages all night. On-call disables the monitor. Two weeks later a real drop is missed. Repair: launch calendar suppresses upper-bound anomalies; keep zero-volume pages; require a human to acknowledge suppressions with an end time. Disabled forever is not a strategy.

Cost monitor as observability

Lesson 58's dry-run byte budget for the certified query is scheduled daily. A BI developer widens the extract to include a join back to raw. Bytes jump ten times. The cost monitor tickets (not pages at midnight) with the query fingerprint. Analytics fixes the extract before the monthly invoice becomes the incident report. Observability includes economic SLIs when the platform's survival depends on them.

Kafka lag without crying wolf

Progress landing consumer lag rises every evening when producers burst and consumers compact. A static lag page fires nightly and is ignored. Replace it with lag growth rate while landing freshness is still meeting its intermediate SLO, and page only when lag predicts missing the compaction window that feeds the mart. Pair lag with landing partition presence. Lag alone is a sensor; freshness of landing is the user-relevant derived signal for this path.

Schema drift in the raw zone

Raw landing is allowed to be wider and messier than the mart. Still ticket on sudden loss of fields the mart maps from, and on type changes that will break Spark jobs tomorrow. Do not page executives for raw schema noise. Do page the ingestion owner if the payload contract from Schema Registry and Event Contracts rejects half the traffic—volume will drop and freshness will follow.

Table stakes for a first observability dashboard

Panels, in order of reading at 05:30:

1. Mart freshness lag and status for today.
2. Yesterday rows versus weekday median.
3. Duplicate groups today.
4. DAG duration versus allowed window.
5. Consumer lag / landing freshness.
6. Open incidents grouped by root asset.

Everything else is a second screen. If the first screen cannot be understood in thirty seconds, the dashboard is a museum, not an operations tool.

Best Practices

• Monitor data and jobs as separate signals; correlate them in the runbook.
• Start with freshness, volume, schema, required nulls, and grain uniqueness on the certified chain.
• Bind every monitor to a lesson 59 owner and a route (page/ticket/dashboard).
• Use weekday baselines and holiday calendars; do not use a single global row threshold.
• Group incidents by root asset; notify downstream, do not necessarily page them all.
• Keep definition_version inside freshness and volume queries.
• Prefer metadata and mart-level SQL over raw full scans for monitors.
• Do not paste PII samples into chat; count and aggregate.
• Treat silent success as a first-class failure mode.
• Export a small set of operational metrics that explain freshness misses.
• Review alert noise weekly for the first month of a new monitor; tune or kill.
• Hand lesson 62 a short SLI list you actually trust, not every panel on the dashboard.

Common Mistakes and Anti-Patterns

Only monitoring job failure. Empty publishes and wrong versions stay invisible. Add data monitors.

Paging on every anomaly. On-call disables everything. Route by user impact and tune baselines.

Monitors without owners. Alerts rot in a shared inbox. Bind lesson 59 names.

Freshness without definition version. You celebrate stale v3 while v4 is empty.

Full-scan raw JSON monitors. You recreate the cost incident. Check marts and partition stats.

Forty pages from one upstream break. Missing lineage grouping. Add root assets.

Dashboard-only for Monday freshness. People sleep. Page for board-threatening freshness.

Schema monitors only in CI. Console DDL bypasses CI. Observe production metadata too.

Slack debug dumps of bad rows containing email. Privacy incident. Use restricted tooling.

Disabling a noisy monitor forever after a launch. Put calendar suppressions with end dates.

Treating consumer lag alone as board health. Lag is a leading indicator; mart freshness is the user SLI.

Troubleshooting and Design Review

Pages fire but the board looks fine. Thresholds may be in the wrong timezone, or monitors read a different definition_version than the board. Align pins and clocks first.

Board is wrong, nothing pages. You are missing uniqueness or version monitors, or the extract is off-contract. Add extract schema checks; do not trust only the mart.

Freshness pages every weekend. Baseline lacks day-of-week logic. Fix the baseline before blaming ingestion.

Platform paged for wrong completion math. Misrouted ownership. Domain owns definition; update alert routes.

Monitor job itself fails silently. Observe the observers: alert when the monitor DAG misses its schedule. Meta-monitoring is dull and necessary.

Lineage says nothing downstream. Graph incomplete. Manually add board SQL and extract edges for certified products before automating the world.

Design review:

• Which SLIs would you take to lesson 62 unchanged?
• What pages on Monday 05:30 versus what tickets?
• Does every page land on a named rotation?
• Can a green DAG still page you on empty publish?
• Do monitors scan the mart or the raw lake?
• How are launch days represented in baselines?
• What is the root asset for progress landing failures?

Interview Questions and Answers

Q1. How is data observability different from APM or infra monitoring?

Infra monitoring tells you whether systems are up and fast. Data observability tells you whether the datasets those systems produce are fresh, complete, schema-correct, and trustworthy. You need both; a healthy cluster can publish empty or wrong-version data.

Q2. What freshness definition would you use for the completion board?

Lag from now to the latest publish_time for the board's event_date and pinned definition_version. The paging threshold is set so humans can still repair before 06:00. I would not use "DAG success time" alone as freshness.

Q3. When do you page versus open a ticket?

Page for user-visible deadline risk and hard correctness breaks (duplicates on grain, required null spikes, schema breaks on certified tables). Ticket for leading indicators, optional column drift, and cost anomalies at night. Dashboard optional exploratory anomalies until tuned.

Q4. How do you detect silent success?

Monitor rows written, partition presence, snapshot advance, and volume baselines even when the orchestrator reports success. Green plus zero rows is a page.

Q5. How does lineage change alerting?

It groups incidents under a root asset and informs who is impacted. Downstream owners get notified; they are paged when their own user-facing SLI fails, not necessarily at the first upstream tremor.

Q6. What is a good first set of monitors on a new mart?

Freshness, volume with weekday baseline, schema versus contract, null rate on required columns, uniqueness on the grain key. Add distribution anomalies later.

Q7. How do you keep observability from exploding the bill?

Run checks on curated tables and metadata; sample for heavy distribution checks; avoid daily full scans of raw JSON; budget monitor compute like any other job.

Q8. How do vendor "data observability" products map to this lesson?

They typically automate baselines, anomaly detection, lineage blast radius, and incident UX. The portable requirements remain: owned assets, SLIs, routing, and certified-path coverage. I can start with SQL monitors and adopt a vendor when the registry maintenance is the bottleneck.

Q9. How do streaming quality monitors interact with the certified mart?

Streaming cursors guard provisional tiles and landing health. Certified mart freshness remains a separate SLI. I never mark the board green because a provisional tile is green.

Q10. What operational metrics help explain a freshness miss?

DAG duration, task failure state, warehouse queue time on the ELT pool, and consumer lag on progress landing. They point to capacity, logic, or upstream silence after the data monitor fires.

Q11. How do you stop launch-day false positives?

Product-owned calendar suppressions for upper anomalies with mandatory end timestamps; keep zero-volume paging active; review suppressions weekly.

Q12. What do you hand to reliability/SLO design next?

A short list of trusted SLIs—Monday freshness success ratio, duplicate-grain rate, certified query failure rate—with current baselines and paging policies already in use.

Hands-On Exercises

Exercise: Observability design for the completion board

Problem statement. Design a data observability dashboard and alert policy for TechLearningPro from progress landing through published.completion_attempt to the Monday board. Include monitor registry entries, SQL or pseudocode for freshness and volume, paging versus ticket rules, lineage grouping, and an empty-publish detection path. Do not depend on a proprietary vendor API.

Tasks.

1. List assets on the certified chain and assign page/ticket owners.
2. Define freshness and volume monitors with thresholds and weekday logic.
3. Add schema, null, and uniqueness monitors with routes.
4. Write SQL (any common dialect) for freshness lag and duplicate grain detection.
5. Specify operational metrics that explain freshness misses.
6. Describe incident grouping when landing volume drops to zero.
7. State how definition_version cutovers avoid false pages.
8. Document what you will not monitor on day one and why.

Expected deliverables. Monitor YAML (or table), SQL snippets, routing policy, lineage grouping note, cutover note, explicit non-goals.

Evaluation rubric.

On-call hygiene for data pages

Pages include the monitor id, asset, value, threshold, owner, and runbook link. Acknowledge within the policy window even if you cannot fix yet. If you suppress, set an end time and a reason (launch_calendar, known_upstream_outage). Suppressions without end times become permanent blind spots. After a Monday incident, add one improvement: a better threshold, a missing monitor, or a runbook step—not a novel.

Comparing provisional and certified signals

Lesson 47's provisional tile may update every few minutes. Its freshness SLI is tighter and its correctness bar is softer. The certified mart SLI is looser in time (batch) and harder on grain. Dashboards that overlay both must label them. Alert routes must not page the certified owner because the provisional tile blipped, or ignore a certified empty publish because the tile still moves. Two products, two monitors, two runbooks.

Warehouse query failure as a synthetic SLI

If the certified SQL fails for the board identity—permission revoked, table renamed, view broken—the board is down even when publish_stats look perfect. A synthetic runner executes the certified SQL every fifteen minutes in the morning window with a byte cutoff and records success, latency, and row count. Page on repeated failure. This catches lesson 57 identity mistakes and lesson 59 cutovers that forgot a grant.

Building the monitor registry without boiling the lake

Week one: certified chain only (landing, compact, mart, board SQL, extract). Week two: add cost dry-run and duplicate grain if not already present. Week three: tune noise. Only then expand to secondary marts. Teams that "enable all table monitors" on day one drown and disable the product. Coverage percentage of certified assets matters more than raw table count.

Incident timeline example

05:12 mart_freshness pages domain. 05:14 runbook shows DAG success, rows_today=0. 05:18 volume confirms empty; landing volume OK. 05:25 root cause: join to empty taxonomy dimension. 05:40 revert deployed via hotfix CI. 05:55 publish_stats shows rows. 05:58 synthetic board SQL green. 06:01 board opens. Post-incident: CI adds fixture for empty dimension; monitor already earned its keep. Without freshness+volume, discovery happens at 06:10 in the meeting.

Duplicate grain incident shape

Unlike empty publish, duplicates often arrive with normal row counts and on-time freshness. Rates look slightly low or wild depending on the join. Uniqueness monitors are the ones that page. Runbook branch: stop incremental merge if it worsens duplicates; compute dup_groups by attempt_id; check whether a replay without idempotent keys double-wrote; republish the day with the idempotent model from earlier modules; invalidate extracts. Capacity restarts do nothing. Lesson 59 change control should have caught a non-idempotent backfill; observability catches it when process fails.

Schema break after console DDL

Someone adds email to the mart "temporarily" in the console. Schema monitor pages on unexpected column and, if classification is wired, privacy is looped in (lesson 61). Volume and freshness stay green. This is why production metadata monitors exist beyond CI. Roll forward by dropping the column through a reviewed change, scrubbing snapshots per privacy guidance, and revoking console DDL rights for certified tables.

Linking cost spikes to observability

A scan-cost ticket at 10:00 Monday shows the board identity scanned raw JSON. Observability marks a "certified path violation" secondary signal: synthetic SQL may still pass if someone runs a second query. Add a monitor on query history fingerprints for the board identity that tickets when scanned bytes exceed the mart budget. It is not a 05:00 page every night; it is a same-morning ticket that prevents lesson 58 from being rediscovered monthly.

What success looks like after thirty days

Paging precision above ninety percent for Monday freshness. Zero silent empty publishes reaching the meeting. Duplicate grain never unmonitored. On-call can name the SLIs without opening a wiki. Monitor registry covers one hundred percent of certified chain assets. Noise review happened every week for the first month and then monthly. That is enough to walk into lesson 62 with honest SLIs.

Anti-pattern: metrics backend without semantics

Exporting five hundred Prometheus metrics from Spark and Airflow feels like observability. Without dataset-level freshness and grain checks, you still ship empty tables. Start from the board's questions—is today's mart here, complete, correctly versioned?—and only then add runtime metrics that explain failures.

Hand-off checklist into reliability design

Before lesson 62, freeze a one-pager: SLI names, current baselines, paging thresholds, owners, and known false-positive rates. Include Monday freshness success, empty-publish detection, duplicate grain rate, and synthetic certified SQL success. Explicitly exclude vanity metrics. Reliability design should negotiate targets against these measured signals, not invent new undefined percentages in a workshop.

Why forty board readers change alerting

Forty concurrent readers do not each need a page when the mart is stale—they need one incident and a status note. Observability groups user reports under the mart freshness incident. Conversely, one reader hitting raw JSON is a cost/governance ticket, not a freshness page. Match signal to blast radius.

Lesson Summary

Observability for data platforms means watching the data's freshness, volume, schema, nulls, and grain—and watching the jobs that explain those signals—then routing pages only when user-visible promises are at risk. TechLearningPro's Monday board needs a freshness SLI with definition_version, a zero-volume page for empty publishes, and owners from DataOps. Lineage groups blast radius. Baselines respect weekdays and launches. Vendors can automate patterns; they cannot invent your ownership or your SLIs.

Data Governance, Privacy and Compliance ensures monitors and debug paths do not become a second privacy hole. Data Reliability and SLA/SLO Design will bind the SLIs you trust to error budgets. Keep the page quiet enough that when it fires, someone still runs.

Further Reading

• Google SRE book — Monitoring Distributed Systems for SLI thinking applied here to data products.
• Great Expectations and similar libraries for expectation-based checks you can schedule as monitors.
• OpenLineage for portable lineage events that support blast-radius grouping.
• Apache Airflow documentation for DAG metrics and alerting hooks.
• Prometheus or your metrics backend docs for exporting job and lag gauges.
• Lessons 17, 22, 47, 58, and 59 for quality, lineage, cursors, cost, and ownership this lesson depends on.
• Conceptual reading on commercial data observability patterns (freshness/volume/schema anomaly monitoring) while keeping implementations vendor-neutral.