Data Reliability and SLA/SLO Design

Data Engineering & Modern Data Platforms · Lesson 62 · Production Data Engineering and Capstone

Lesson Overview

The board is late and nobody can say by how much. Forty people open the Monday completion board at 06:00. The tile loads. The number is yesterday's. Someone asks whether the pipeline failed, whether the warehouse is slow, or whether the definition changed. Three chats open. Nobody owns a sentence that says what "on time" means, how often it may miss, or which clock you are allowed to blame.

Data Platform Cost and Performance Optimization priced the knobs that keep this board affordable. It also handed Module 6 a refusal list: do not drop the grain check, do not serve a stale definition_version, do not pause capacity when resume misses 06:00. This lesson turns those refusals into a contract. An SLA is the promise you make outside the team. An SLO is the measurable target you operate to. An error budget is how much miss you can afford before you stop shipping features and start repairing reliability. Without those three, every invoice debate and every incident ends in opinion.

The teaching volumes stay fixed. About 40 GB of compacted Parquet progress a day. A completion mart that is tens of gigabytes once it is columnar. About forty board readers on Monday morning. The certified promise is that the Monday board shows yesterday's completion grain, with definition_version pinned, by 06:00 local. A provisional tile may move faster on a different cursor. It does not redefine the certified clock. Data Observability and Operational Monitoring supplies the signals. This lesson decides which signals are the SLO, how you burn the budget, and what you refuse to trade when the cost sheet gets loud.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate SLI, SLO, SLA, and error budget with sentences a product owner can sign.
2. Write a freshness SLO for the TechLearningPro completion board that names the clock, the grain, and the deadline.
3. Distinguish pipeline latency from query latency so a slow dashboard does not look like a late merge.
4. Choose completeness, correctness, and availability indicators that do not double-count the same failure.
5. Compute an error budget in missed mornings and in minutes late, and say when each form is useful.
6. Apply burn-rate and multi-window alerting so a single late night does not page, and a week of near-misses does.
7. Set recovery objectives (RPO/RTO) for the mart that match restore drills, not slideware.
8. Refuse cost cuts that spend the freshness budget, and name the cuts that are still allowed.
9. Design an escalation path when the budget is exhausted: freeze risky changes, keep the grain check, reopen only with a written plan.
10. Hand Data Pipeline Deployment and Infrastructure as Code a reliability sheet that CI must protect, not a hope in a runbook.

Prerequisites

• Data Platform Cost and Performance Optimization for the meters and the refusal list this SLO constrains.
• Data Observability and Operational Monitoring for freshness, volume, and schema signals. This lesson selects which of those become the contract.
• DataOps and Data Platform Engineering for ownership and decision records.
• Data Governance, Privacy and Compliance when a late board would still be better than a wrong board that leaks.
• The lakehouse, Iceberg or Delta, and cloud serving lessons for publish semantics: a commit is the moment the board is allowed to see a day.
• Orchestration and testing lessons so a failed DAG and a green DAG with wrong grain are both reliability events.

Detailed Core Concepts

SLI, SLO, SLA, and the sentence that stops the chat

An SLI (service level indicator) is a measured ratio or absolute: fraction of days the mart was published by 06:00 with the pinned definition; p95 query time for the certified board SQL; fraction of progress events that pass the grain check before publish. An SLO is the target on that indicator: 99% of business mornings meet the publish, or fewer than two missed Mondays per quarter. An SLA is the external or contractual consequence: credits, priority response, a written escalation to the product VP. Most internal data products have an SLO and a clear owner long before they have a legal SLA. Do not invent an SLA to look serious. Invent an SLO you can measure tomorrow.

The sentence that stops the Monday chat is not "the pipeline is usually fine." It is:

Certified completion for event_date = yesterday, definition_version = completion-v4, is queryable on the serving mart by 06:00 local on business mornings, for at least 99% of those mornings in a rolling 90 days.

That sentence names the product, the grain, the version, the clock, the window, and the target. Everything else in this lesson hangs from it.

Freshness is not one number

Freshness for this platform has at least three clocks.

Event time is when the learner action happened. Ingest time is when the event landed in the raw zone. Publish time is when the serving table's commit made that day's grain visible to the board identity. The board's freshness SLO is about publish time relative to a wall clock (06:00), not about how long a single event waited in a topic. A streaming tile that shows "last five minutes" is a different product with a different SLO. Mixing them produces a false incident: the certified merge is on time, the provisional tile is behind, and someone pages the wrong owner.

Pipeline latency is the span from the close of the day's ingest window to publish. Query latency is the span from a reader issuing the certified SQL to a result. A warehouse that is cold, a BI extract that is stale, or a dashboard that scans raw JSON can make the board feel late when the merge finished at 05:20. Measure both. Alert on both. Do not merge them into "the data is slow."

Completeness and correctness are different failures

Completeness asks whether the day is whole enough to publish: expected volume bands, partition presence, watermark past the day boundary, duplicate rate under a threshold. Correctness asks whether the published grain means what the product says: definition_version matches the contract, forbidden columns are absent, joins did not inflate attempts, refund or completion rules match the signed definition.

A complete wrong day is worse than an incomplete late day for a board that drives bonuses. Completeness SLOs without a correctness gate encourage shipping garbage on time. Correctness gates without a freshness budget encourage silent freezes. Pair them: you may miss 06:00 to finish the grain check; you may not skip the grain check to hit 06:00. That pair is the reliability version of lesson 58's refusals.

Availability is the serving path, not the job green check

Availability for the board is: the certified query returns successfully within a latency budget for the board identity, against the published table, during the morning window. A green Airflow DAG with a private table that the BI service account cannot read is an availability miss. A paused warehouse that resumes after 06:05 is an availability miss even if publish finished at 05:50. Orchestration success is necessary and not sufficient.

Error budgets turn reliability into a decision, not a feeling

If the SLO is 99% of business mornings on time over 90 days, and there are about 65 business mornings in 90 days, the error budget is roughly 0.65 mornings: fewer than one full miss per quarter if you spend it in whole days, or a few partial misses if you count minutes. Teams that only count binary miss/hit will hide chronic 06:12 publishes as "success." Prefer a primary indicator that is binary for the board (ready by 06:00 or not) and a secondary that records minutes late when it misses, so postmortems have a distribution.

When the budget is healthy, you may ship DAG changes, schema expansions, and cost experiments that lesson 58 allows. When the budget is burned, you freeze risky deploys, keep the checks, and spend engineering on the top burn causes. The budget is the policy that connects observability to change management. Without it, every late morning is a debate about whether to "move fast."

Burn rate and multi-window alerts

A burn rate compares how fast you are consuming the budget to how fast you are allowed to. Burning the entire quarterly budget in two days is a page. Burning one morning in ninety days is a ticket. Multi-window alerting, adapted from Google's SRE practice for request SLOs, applies here with calendar windows instead of request counts:

• Fast window: last 2 business days. High burn → page if both windows agree you are on track to exhaust the budget soon.
• Slow window: last 14 or 30 business days. Elevated burn → ticket, not page, unless the fast window is also on fire.

A single Monday miss should not wake people if the quarter still has budget and the cause is understood. A pattern of 06:08 publishes that never trip a binary miss still burns a minutes-based budget and should open a ticket before the next quarter starts empty.

Recovery objectives are not freshness

RPO (recovery point objective) is how much data you can afford to lose in a restore: for the mart, usually "replay from raw for the affected days" rather than "zero event loss," because raw is the system of record for rebuild. RTO (recovery time objective) is how long until the board is usable again after a bad publish or a deleted table. A freshness SLO of 06:00 does not imply RTO of minutes. State them separately. A restore drill that takes four hours is fine for a rare disaster if the product accepts "board offline until rebuild finishes," and catastrophic if the SLA promised same-morning recovery. Write the honest one. Lesson 58 already refused unpaid cross-region copies without a restore test. This lesson is where that test gets an RTO number.

What you refuse to trade for cost

Lesson 58 listed refusals. Restate them as SLO policy:

• Do not remove the grain check or prohibited-column gate to buy minutes.
• Do not pause board capacity when measured resume misses 06:00.
• Do not point the board at raw JSON to avoid a warehouse bill.
• Do not serve a cached extract across a definition_version change.
• Do not let the provisional five-minute tile write the certified table.

Cost cuts that remain allowed: compact small files, prune columns, isolate ELT from the board pool, lower idle capacity outside the SLO window after resume is timed, drop a second mart, turn off untested replicas. Reliability is the constraint. Cost is optimized inside it.

SLAs you should not invent

Internal platforms often do not need a legal SLA. What they need is an SLO, an owner, a communication channel when the budget burns, and a product escalation when the deadline itself must move. Inventing "SLA credits" for an internal board confuses finance and does not improve Monday. If an external customer contract requires an SLA, map it to the same SLIs and keep the internal SLO at least as tight as the external promise. Never run a looser internal target than the contract you signed—the contract will win in the war room.

Calendar realities: holidays, DST, and half days

Business mornings are not "all Mondays–Fridays in the timezone of the warehouse." They are the mornings the board meeting exists. Publish a calendar: exclude company holidays, include rare Sunday launches if product runs them, and decide what happens when clocks change. DST spring-forward can delete an hour from the critical path; fall-back can duplicate a local hour in logs. Store published_at in UTC, express the deadline in a named local zone, and test the boundary days once per year. A freshness miss caused by an untested DST transition is still a miss—and a process miss in the design review.

Human SLOs: ownership and response

A data SLO without a human response target becomes a graph nobody pages on. Define: who is primary on-call for freshness P1; acknowledge within N minutes; update the board channel with "investigating / mitigated / excluded" language; post a short write-up for any miss that burned budget. This is not theater. Lesson 60 gives signals; humans close the loop. If the team is too thin for 24/7, say the coverage window explicitly (for example 05:00–10:00 local on business mornings) so expectations match staffing.

Architecture and Design Diagrams

The board's reliability path is a chain of clocks. Any break can burn the budget; only some breaks are the merge's fault.

mermaidCopy
flowchart LR
  EVT["Event time"] --> ING["Ingest / raw"]
  ING --> XFORM["Validate and transform"]
  XFORM --> PUB["Publish commit"]
  PUB --> SERVE["Serving mart"]
  SERVE --> Q["Certified query"]
  Q --> BOARD["Monday board"]
  PUB --> FRESH["Freshness SLI"]
  Q --> AVAIL["Availability / query latency SLI"]
  XFORM --> CORR["Correctness gates"]

Error budget as a policy loop, not a dashboard wallpaper:

mermaidCopy
flowchart TD
  SLO["Freshness SLO 99% by 06:00"] --> BUDGET["Error budget"]
  BUDGET --> HEALTHY["Budget healthy: ship with checks"]
  BUDGET --> BURN["Burn rate high: ticket or page"]
  BURN --> FREEZE["Freeze risky changes"]
  FREEZE --> FIX["Repair top burn causes"]
  FIX --> BUDGET
  HEALTHY --> COST["Cost knobs allowed by lesson 58"]

Pipeline latency versus query latency, so alerts route correctly:

mermaidCopy
sequenceDiagram
  participant Dag as Merge DAG
  participant Mart as Serving mart
  participant Bi as Board query
  participant User as Reader
  Dag->>Mart: Commit day D at 05:25
  Note over Mart: Freshness SLI satisfied for 06:00
  User->>Bi: Open board 06:02
  Bi->>Mart: Certified SQL
  Mart-->>Bi: 45s (cold capacity)
  Bi-->>User: Tile late to eye
  Note over Bi,User: Query latency SLI miss, not publish miss

Technical Explanation

Writing the freshness SLI so it can be computed

Prefer an explicit publish marker over inferring from "max(event_date) in the table." A marker row or a status table records event_date, definition_version, published_at, row_count, and pipeline_run_id. The SLI for a business morning M is true when there exists a marker for event_date = M - 1 day with the pinned version and published_at <= M at 06:00. Inference from table contents fails when a partial load wrote some partitions, or when a backfill of an old day updates max(event_date) while yesterday is missing.

sqlCopy
-- Illustrative status grain. One row per published day and definition.
SELECT event_date,
       definition_version,
       published_at,
       row_count,
       pipeline_run_id
FROM ops.publish_status
WHERE event_date = DATE '2026-11-10'
  AND definition_version = 'completion-v4';

The observability lesson may scrape job end times. Job end time is a good secondary. The marker is the primary, because the board reads the table, not the DAG UI.

Pipeline latency budget inside the morning

Work backwards from 06:00. Leave margin for resume, for the board's first refresh, and for a short retry.

If gates regularly finish at 05:58, you have no retry budget. Move the merge earlier or cut work that is not on the critical path. Do not cut the gates.

Query latency SLO is a different contract

A reasonable board query SLO for forty concurrent readers of the pruned mart: p95 under 10 seconds for the certified SQL during 06:00–10:00, availability 99.9% of requests in that window. That number is a teaching default. Replace it with measurements. What matters is separation: a publish at 05:40 with p95 of 90 seconds is a serving problem. A publish at 06:20 with p95 of 3 seconds is a pipeline problem. Combined "user waited" metrics are fine for product, useless for routing the repair.

Completeness bands without fake precision

Set volume bands from history: yesterday's progress row count should lie within, say, 60% to 140% of the same weekday's median over eight weeks, unless a product calendar says otherwise. Outside the band, do not auto-publish. Page or ticket based on burn policy. Absolute counts without seasonality will false-alarm every holiday and miss a half-day outage on a quiet Wednesday.

Duplicate rate and null-critical-field rate are correctness-adjacent completeness checks. Decide thresholds that match the definition: a 0.1% duplicate rate after idempotent merge may be acceptable noise; a 5% null lesson_id is not a publish.

Burn-rate arithmetic you can explain on a whiteboard

Let B be the error budget in missed mornings per quarter (for example 1.0). Let W_fast be 2 mornings and W_slow be 14 mornings. If both mornings in the fast window missed, you burned 2 / B of the quarterly budget in two days. That is a high burn: page. If 3 of the last 14 missed, burn is elevated: ticket, review the top cause, freeze non-essential DAG changes until the slow window recovers. Exact thresholds belong in the reliability sheet. The shape matters more than the constant: two windows, two severities, one budget.

pythonCopy
"""Illustrative burn helpers. Replace thresholds with the signed sheet."""

def missed_fraction(misses: int, windows: int) -> float:
    return misses / windows if windows else 0.0


def burn_multiple(misses: int, window: int, budget_fraction_per_day: float) -> float:
    """How many times faster than allowed we are burning in this window."""
    allowed = budget_fraction_per_day * window
    if allowed <= 0:
        return float("inf")
    return misses / allowed


# Example: 1 miss allowed per 65 business days ≈ 0.0154 per day
BUDGET_PER_DAY = 1.0 / 65
fast = burn_multiple(misses=2, window=2, budget_fraction_per_day=BUDGET_PER_DAY)
slow = burn_multiple(misses=3, window=14, budget_fraction_per_day=BUDGET_PER_DAY)
print(f"fast_burn={fast:.1f}x slow_burn={slow:.1f}x")

Multi-signal incidents without alert storms

One failed publish should open one incident with children: freshness miss, gate failure reason, capacity state. Do not page separately for "DAG failed," "marker missing," and "board 500" as three unrelated fires when they share a root. Lesson 60's correlation is the mechanism. This lesson's contribution is priority: freshness and correctness of the certified product outrank a failed exploratory job in the same MWAA environment.

Dependency SLOs and blame

If progress events stop in the application API, the data platform will miss publish. The honest model is a dependency SLO: platform freshness assumes ingest volume above a floor by 04:00. Below that floor, the incident is shared or owned by the producer, and the platform's error budget may exclude that morning if the contract says so. Write the exclusion. Silent exclusions that appear only in postmortems destroy trust in the budget.

Minutes-late budgets versus binary mornings

Binary success by 06:00 is the right primary for a standup board: either the meeting has today's number or it does not. A minutes-late secondary catches the team that "succeeds" at 05:59 every day with no retry margin, then fails hard on the first skew. Track max(0, published_at - 06:00) on misses, and also track 06:00 - published_at on hits as margin. Margin trending from forty minutes to three minutes is a slow burn even when the binary SLI stays green. Put margin on the weekly reliability review. Do not page on margin alone unless the product asks for it.

Some organizations convert everything to "good minutes" in a month: each morning contributes 1 if on time, or 1 - late_minutes/allowed_grace if within a short grace, else 0. Grace is dangerous if it becomes the real SLO. If you use grace, cap it at a few minutes and still count a grace publish as budget spend at a fractional rate. Otherwise the culture relocates the deadline without a product conversation.

Backfills, replays, and the certified clock

A backfill of last month must not move yesterday's freshness SLI. Markers are keyed by event_date and definition_version. Replaying day D updates the marker for D and may invalidate extracts for D; it does not satisfy the morning obligation for day D+1. Orchestration that uses a single "pipeline succeeded" flag for all dates will lie during backfill weeks. Require the morning checker to read the marker for the specific event_date the board needs.

Idempotent replay is a reliability feature when a bad publish must be repaired inside RTO. It is a reliability hazard when two writers can publish the same day with different definitions. The deploy lesson will require one publisher identity. The SLO assumes that identity. If a human merges SQL by hand in the warehouse console, the marker contract is broken even if the clock looks fine.

Partial publish and degraded modes

Product may ask for a degraded board: publish what passed gates for 90% of lessons, mark degraded=true, and show a banner. That is a different SLO path. Encode it explicitly: degraded publish may count as freshness success only if the banner is mandatory and correctness for the included slice still passes. Never treat a silent partial load as success. Northline-style finance tables usually forbid degraded mode. TechLearningPro's internal board may allow it for a documented class of upstream gaps. Write the class down. Ambiguous degradation is how wrong numbers become "on time."

Linking SLOs to change management and CI

The reliability sheet is an input to deployment policy. CI should be able to answer: does this change touch gates, publish markers, board SQL, or capacity schedules? If yes, and budget remaining is below a threshold, require a reliability owner approval. Lesson 63 implements that as branch protection and environment promotion. Lesson 62 defines the threshold and the owner. A sheet that lives only in a wiki slides under the first urgent feature.

textCopy
reliability_gates:
  freshness_slo: "99% business mornings publish by 06:00"
  budget_window_days: 90
  freeze_below_remaining_pct: 25
  protected_paths:
    - dags/completion/**
    - sql/published/**
    - gates/**
    - terraform/warehouse_schedule/**

The YAML is illustrative. The idea is durable: protected paths, freeze threshold, named SLO.

Worked example: one bad week

Assume 65 business mornings per 90 days and a budget of one full miss (≈1.5% miss allowed). Week starts with full budget. Monday publish 06:18 (miss). Tuesday 05:50 (hit, margin 10). Wednesday gate fail, no marker (miss). Thursday 05:40 (hit). Friday 06:02 (miss if deadline is strict 06:00).

Misses: three in five days. Fast-window burn on Thursday–Friday already high. Slow window will show elevated burn. Policy: page on the third miss if fast burn crosses the sheet's multiple, freeze DAG feature work, keep gates, inspect critical path. Root cause pattern in this sketch is often shared pool contention on Mon/Wed and an overnight pause that resumed late on Friday. The cost review's pause experiment is now evidence-backed as a budget burner. Restore capacity schedule; do not delete the Wednesday gate that correctly blocked a bad grain.

Practical Implementation or Design Walkthrough

Problem

TechLearningPro needs a signed reliability sheet for the completion board before the next cost review tries to pause the warehouse overnight and before a new DAG author removes a slow grain check. Produce the SLIs, SLOs, error budget, alert windows, and refusals.

Step 1: Freeze the product sentence

textCopy
Product: Certified Monday completion board
Grain: one row per learner-lesson attempt per event_date (published definition)
Version pin: completion-v4
Audience: ~40 internal readers, business mornings 06:00–10:00 local
Not in scope: provisional "last 5 minutes" tile (separate cursor and SLO)

Step 2: Define SLIs

Step 3: Set SLOs and budget

Correctness at 100% of marked publishes means you fail closed: no marker if gates fail. You do not mark ready and hope.

Step 4: Wire alerts

textCopy
Page: freshness miss on a business morning AND (fast-window burn high OR correctness gate failed after a retry)
Ticket: slow-window burn elevated OR p95 latency above SLO for 3 consecutive mornings
Info: single near-miss (published 05:55–06:00) for trend review

Step 5: Time the critical path once

Run a game day on a Tuesday. Record watermark, compaction, merge, gates, marker, resume, and first query. If marker lands at 05:52 and resume ends at 06:01, the cost sheet may not pause that capacity. The reliability sheet says so explicitly, citing the measurement date.

Step 6: Write refusals into the change policy

textCopy
While freshness budget < 25% remaining:
  - No DAG changes that remove or skip gates
  - No warehouse pause experiments
  - No definition_version cutovers without dual-run
Normal budget:
  - Cost knobs from lesson 58 allowed with dry-run evidence
  - Schema additive changes behind expanded gates

Step 7: Document dependency exclusions

If ingest volume by 04:00 is below 50% of the weekday floor, classify as producer outage. Platform still attempts partial publish only if product asks; default is no marker and a joint incident. Budget exclusion requires a linked producer incident id.

Step 8: Publish the sheet where CI can read it

Store config/reliability.yaml next to the DAG package. Fields: product sentence id, freshness deadline, budget window, freeze threshold, protected paths, producer floor, query p95 target, owner aliases. Lesson 63's promote job will parse the freeze threshold. A PDF in a slide deck cannot freeze a deploy.

Step 9: First operational week

Day 1: enable marker writes in prod alongside existing jobs without enforcing yet; compare marker time to DAG end time. Day 2: alert in ticket-only mode on marker miss. Day 3: page on miss if after the deadline and not excluded. Day 4: add slow-window burn ticket. Day 5: review margin trend with the cost owner present so pause debates use the same numbers. Do not enable pages on day 1 with untrusted markers; you will train the org to ignore you.

Worked arithmetic card (bring to the cost review)

textCopy
Business mornings in next 90 days: 64 (example; count the calendar)
Freshness SLO: 99% → allowed misses: floor(64 * 0.01) = 0  if you truncate,
  or use 99% as "expect ≤ 1 miss" and state it as absolute: 1 miss / 64 mornings
Choose absolute misses for humans: budget B = 1 morning
Minutes-late secondary: sum of late minutes on misses; review if any hit margin < 10 min
Query p95 target: 10s (replace with measured baseline + 50% headroom)
Resume probe: must complete ≤ 8 min or board_pause_enabled=false

Label every constant as measured or proposed. The reliability sheet is allowed to say "proposed pending Tuesday game day." It is not allowed to say "TBD" on the deadline itself.

Enterprise Real-World Scenarios

TechLearningPro's silent 06:12

For three weeks the merge finishes between 06:05 and 06:15. The binary "failed" alert never fires because someone set the threshold to 06:30 "to reduce noise." The board meeting starts at 06:00 with yesterday's cache. Leadership concludes the data team is unreliable. The cost review proposes deleting the grain check to save fifteen minutes.

The repair is not a louder page for every job. Redefine the SLO at 06:00, count each late morning against the budget, and open a ticket on the slow window after the second week. Move compaction earlier, isolate ELT, keep the gate. Show the cost review the refusal list and the measured critical path. If the only way to hit 06:00 is skipping correctness, the product must choose a later board or a narrower grain. That choice is executive. It is not an engineer's silent edit to the alert threshold.

Northline's on-time wrong close

Northline's finance close table arrives every morning by 07:00. Refunds for status code REVERSED were dropped after a "performance" SQL change. Completeness bands still passed because volume stayed in range. The freshness SLO was green for a month while revenue was wrong.

They add a correctness SLI: reconciliation of refund amounts to the payment export within a tolerance, and a canary comparison of definition_version aggregates day over day. Freshness may miss when reconciliation fails. The error budget for freshness is allowed to burn to protect correctness. Finance prefers a two-hour delay over a restatement. The SLA with finance says so. The platform stops treating "green DAG" as done.

TechLearningPro dual-cursor confusion

Product launches a "live progress" tile fed from a five-minute micro-batch into a landing table. On-call pages when that tile lags, while the certified board is healthy. Engineers pad the certified merge to help the tile, miss 06:00, and burn the real budget for a product that never had a signed five-minute SLO.

Split the contracts. Provisional tile: lag SLO from event time to landing visibility, separate runbook, separate alert route, no write path into published.completion_attempt. Certified board: unchanged 06:00 marker SLO. Cost and capacity for the tile are a different line on the lesson 58 sheet. Reliability ownership follows the sentence, not the shared Airflow environment name.

Capacity pause versus freshness after a "successful" savings quarter

Finance celebrates a lower warehouse bill after nightly pause. Three near-misses become two hard misses when resume times drift from four minutes to eighteen after a platform upgrade. The savings were real. The error budget is gone. The reliability sheet should have required a resume probe in CI or in a synthetic morning check after any change to schedules, instance types, or network policies. A cost win without a resume SLI is an unpriced risk transfer onto Monday's meeting.

Best Practices

• Write one product sentence with grain, version, clock, and audience before any percentage.
• Use an explicit publish marker as the freshness primary.
• Separate pipeline latency from query latency in SLIs and in routing.
• Fail closed on correctness: no marker when gates fail.
• Give correctness a zero budget for shipping known wrong data; spend freshness budget instead.
• Use multi-window burn alerts so one miss is not a page storm and chronic near-misses are not invisible.
• Measure the critical path before approving overnight pauses or gate deletions.
• Name producer dependency floors and budget exclusions in the sheet.
• Keep provisional tiles on a different SLO and a different table.
• Tie cost experiments to remaining error budget; lesson 58 knobs are privileges of a healthy budget.

Common Mistakes and Anti-Patterns

An SLO of "as soon as possible." It cannot burn, cannot alert, cannot refuse a cost cut. Replace with a clock.

Alerting on DAG success only. The board reads tables and query APIs. Marker and query SLIs are the product.

Raising the freshness deadline to silence pages. You moved the goal. You did not improve reliability. Change the product commitment in public if you must move it.

One combined "user latency" SLI for publish and query. You will restart the warehouse for a late merge and rerun the merge for a cold cluster.

Volume checks without weekday seasonality. Holidays page. Quiet outages do not.

100% freshness SLO with no budget. Every miss is a crisis and every change is blocked, or the SLO is quietly ignored. Prefer 99% with a real budget unless the product truly needs zero misses and pays for the engineering.

Spending correctness to buy freshness. Restatements cost more than a late tile. Fail closed.

Provisional streaming path writing the certified mart. Two cursors, one table, undefined SLO. Separate sinks.

Error budget as a dashboard without a change policy. Numbers that do not freeze risky deploys are decoration.

Excluding every miss as a "dependency" after the fact. Pre-write exclusion rules. Retroactive exclusions empty the budget's meaning.

Troubleshooting and Design Review

Board looks stale, DAG green. Check the marker, the BI extract's definition_version, and query latency. Likely a cache or a view pinned to an old version.

Marker late, gates passed. Inspect queue time on the shared pool, small files, upstream watermark. Critical path timing from the last game day is the baseline.

Pages every Monday, quiet otherwise. Monday volume or resume timing; not a generic "Airflow is unstable." Adjust capacity schedule, not only retries.

Budget empty mid-quarter. Freeze non-essential changes, list top burn causes by count of missed mornings, schedule repairs before feature work. Do not reset the budget without a signed product change to the SLO.

Query SLO red, freshness green. Serving path: capacity, concurrency, SQL regression, file layout. Do not rerun yesterday's merge as the first step.

Freshness green, finance disputes the number. Correctness path: reconciliation, definition drift, double mart. Freshness tooling will not find it.

Design review questions:

• What is the exact product sentence?
• Where is the publish marker written and who can forge it?
• What is the remaining error budget and what freezes when it is low?
• What is the measured critical path to 06:00?
• Which lesson 58 knobs are currently allowed?
• What producer floors exclude a morning from the platform budget?
• Is the provisional tile isolated?

Interview Questions and Answers

Q1. How do you define a freshness SLO for a daily board?

I name the grain, the definition version, the serving location, and the wall-clock deadline. The SLI is usually an explicit publish marker for yesterday's date with that version at or before the deadline. I set a target like 99% of business mornings over a rolling window and derive an error budget. I do not use "DAG success" alone, and I do not mix a real-time tile's latency into the same SLO.

Q2. What is an error budget for a data pipeline?

It is the allowed amount of SLO miss over a window, expressed so the team can make change decisions. For a 99% morning freshness target, the budget is the remaining percentage of mornings. When burn is high, we freeze risky deploys and fix reliability. When budget is healthy, we may ship changes and approved cost experiments. Without a budget, every miss is either ignored or treated as a catastrophe.

Q3. How do burn rate and multi-window alerting apply to batch data products?

Burn rate compares misses in a short window and a long window to the allowed miss rate. A fast window catching multiple misses pages because the quarterly budget will empty quickly. A slow window with moderate burn opens a ticket. A single miss with low burn is documented without waking the company. The windows are business mornings, not HTTP requests, but the policy shape is the same.

Q4. How do you separate pipeline latency from query latency?

Pipeline latency ends at publish: marker written, table committed. Query latency starts when the board issues SQL or opens an extract. I give each an SLI. A cold warehouse shows up in query latency. A late merge shows up in freshness. Routing the first response to the right system saves the morning.

Q5. When would you miss a freshness SLO on purpose?

When a correctness gate fails. Shipping a wrong definition_version or a broken grain is worse than a late marker for this board. I burn freshness budget, fail closed, and escalate. I do not skip the gate to keep a green freshness chart.

Q6. What do you refuse to trade when finance asks to cut cost?

I refuse to remove grain and column gates, to pause capacity when resume misses the deadline, to point the board at raw scans, and to serve caches across definition changes. I allow compaction, pruning, isolation of ELT, dropping a second mart, and pausing only after a timed resume. Reliability sets the feasible region. Cost optimizes inside it.

Q7. How do RPO and RTO relate to a 06:00 freshness SLO?

They answer a different question: after corruption or deletion, how much can you lose and how fast must you be back. Freshness assumes the happy path. RPO for this mart may be "rebuild affected days from raw." RTO may be hours. I do not claim RTO of five minutes because the board is due at 06:00. I schedule restore drills that measure the real RTO.

Q8. Why is a publish marker better than MAX(event_date)?

MAX(event_date) can move because of a backfill of an old day while yesterday is empty, or a partial write. A marker is an explicit claim that a day and version are certified, with a timestamp and run id. Forgery is a control problem, which I address with job identity permissions. Ambiguous maxima are a design problem.

Q9. How do you handle upstream outages in the error budget?

I predefine dependency floors, such as minimum ingest by 04:00. Below that, the morning is a joint or producer-owned incident, and the sheet states whether it excludes from the platform budget. I require a linked incident id for exclusions. I do not invent exclusions in the postmortem to protect a chart.

Q10. What is wrong with a 100% freshness SLO?

It implies zero budget. Teams either freeze all change, silently ignore the SLO, or move the deadline. A 99% target with an explicit budget creates better behavior: prioritized repairs, honest alerting, and room for controlled change. Some safety-critical feeds truly need stricter targets; they also need the staffing and architecture those targets imply.

Q11. How would you SLO a real-time provisional tile differently?

I would measure end-to-end lag from event time to tile visibility, with a target like 99% of events visible within five minutes, and I would forbid that path from writing the certified daily mart. Availability and correctness still matter, but the clock and the sink differ. One sentence per product.

Q12. How do backfills interact with a daily freshness SLO?

Backfills update markers for the dates they rebuild. The morning checker must request yesterday's marker explicitly. A successful backfill of last month does not make this morning fresh. I also watch that backfill compute cannot starve the certified merge; isolation from lesson 58 applies to reliability as much as to cost.

Q13. What is a degraded publish mode and when is it acceptable?

A degraded publish marks a partial but honest dataset with an explicit flag and UI banner, under rules product signed in advance. It may count toward freshness only if those rules say so. Silent partial loads are incorrect publishes. Finance-grade tables often forbid degradation entirely.

Q14. How does remaining error budget change deployment policy?

Below a threshold such as 25% budget remaining, changes to gates, board SQL, markers, and capacity schedules need reliability approval or are blocked. Healthy budget allows normal delivery and approved cost experiments. The policy must be enforced in CI and promotions, not only in a retrospective slide.

Hands-On Exercises

Exercise: Reliability sheet for the completion board

Problem statement. Design the reliability contract for TechLearningPro's certified completion board. Use the teaching volumes and the 06:00 promise. Incorporate lesson 58 refusals. Do not invent a legal SLA unless you label it as optional.

Tasks.

1. Write the product sentence (grain, version, clock, audience, out of scope).
2. Define five SLIs with sources, including freshness and query latency as separate indicators.
3. Set SLOs and an error budget for 90 days; show the arithmetic.
4. Propose multi-window burn alerts with page versus ticket.
5. Draft a critical-path timetable and state whether overnight pause is allowed.
6. List cost knobs that are allowed versus refused at low budget.
7. Define one producer dependency exclusion rule.
8. Sketch the publish marker schema and who may write it.

Expected deliverables. Reliability sheet (one to two pages), marker DDL or schema, alert policy, critical-path table, refusal list.

Evaluation rubric.

Lesson Summary

Reliability for the completion board is a signed sentence: yesterday's grain, pinned definition, queryable by 06:00 on enough business mornings to leave a small error budget. Freshness, completeness, correctness, and query behavior are different indicators. Pipeline latency and query latency route to different fixes. Burn-rate and multi-window alerts turn misses into policy instead of noise or silence. Correctness fails closed even if freshness burns. Cost optimization from lesson 58 is allowed only inside that feasible region.

Data Pipeline Deployment and Infrastructure as Code is next: the gates, markers, and environment separation must be deployed as code so a hand click in prod cannot quietly delete the contract. Data Engineering System Design will ask you to propose the whole platform under interview pressure with these SLOs as constraints, not as afterthoughts.

Further Reading

• Google SRE Book — Service Level Objectives for SLI/SLO/SLA vocabulary and error budgets.
• Google SRE Workbook — Alerting on SLOs for burn rate and multi-window alerting patterns adaptable to batch mornings.
• Apache Airflow Documentation for SLA miss callbacks as orchestration signals, not as substitutes for publish markers.
• Great Expectations or equivalent gate tooling docs for fail-closed correctness checks before publish.
• Cloud query-history documentation for your serving engine (Redshift, BigQuery, Snowflake, Synapse) to measure query latency SLIs.
• Data Platform Cost and Performance Optimization for the cost refusals this SLO enforces.
• Data Observability and Operational Monitoring for the signal layer beneath the contract.