DataOps and Data Platform Engineering

Data Engineering & Modern Data Platforms · Lesson 59 · Production Data Engineering and Capstone

Lesson Overview

The Monday board is late because nobody owned the change that made it late. A domain engineer merged a new column into completion-v4 on Friday. The DAG still ran. The mart still published. Forty people opened a dashboard at 06:00 and saw a grain that no longer matched the tile. The platform team was paged for a warehouse outage that was not an outage. The definition had moved under the readers. The runbook said "restart the cluster." Restarting the cluster did nothing useful.

Modules 1 through 5 built the path: ingestion, Spark, Kafka, lakehouse, cloud engines, security identities, and the cost sheet that prices idle capacity and raw scans. Module 6 operates that path. This lesson is the operating model. DataOps is not a product name and not a synonym for Airflow. It is the set of practices that make a data change reviewable the way a service change is reviewable: ownership, environments, continuous integration for pipelines and tables, promotion rules, and runbooks that name a person and a table instead of a hope.

TechLearningPro's teaching volumes stay fixed. About 40 GB of compacted Parquet progress a day. A completion mart that is tens of gigabytes once it holds a grain. About forty board readers on Monday morning. A publish that must finish with margin before 06:00 with definition_version pinned. Data Platform Cost and Performance Optimization priced the knobs. Data Platform Security and Governance chose identities. This lesson decides who may turn those knobs, in which environment, and how a table change reaches production without a hero.

The next lessons will observe the data and encode privacy controls. Data Reliability and SLA/SLO Design will write the 06:00 promise as an SLO. You should arrive there with an ownership map and a change gate, not with a chat thread that starts at 05:55.

Learning Objectives

By the end of this lesson you will be able to:

1. Define DataOps as ownership, CI, environments, and change management for data products, not as a tool brand.
2. Separate platform-team responsibilities from domain-team responsibilities for TechLearningPro's progress and completion mart.
3. Name an owner, a backup, and a pager path for every certified table and every production DAG.
4. Design environment promotion (dev → staging → prod) that includes schema, data samples, and orchestration, not only code.
5. Write CI checks that fail a merge when a grain check, a forbidden column, or a contract test breaks.
6. Treat table DDL, Iceberg or Delta commits, and dbt models as change-controlled artifacts with review.
7. Build a runbook that diagnoses a late board by publish cursor and definition version before touching capacity.
8. Apply change windows and freeze rules that protect Monday 06:00 without blocking all work.
9. Decide when a domain may self-serve a new column and when the platform must own the publish path.
10. Hand lesson 60 a list of owned SLIs and lesson 61 a classification owner for PII columns.

Prerequisites

• Lessons 54 through 56 for the single serving engine and the reopen conditions.
• Lesson 57 for identities and audit scope; this lesson assigns people to those identities.
• Lesson 58 for the cost sheet and the refusal list tied to freshness; DataOps enforces those refusals as gates.
• Data Pipeline Testing and CI/CD for test shapes; this lesson places them in an operating model.
• Apache Airflow Architecture and DAGs and Advanced Airflow Operations for orchestration boundaries.
• Data Lineage and Metadata Management for catalog entries that ownership will hang on.
• Schema Registry and Event Contracts for contract tests that belong in CI.
• Lesson 62's place in the course, so a change that breaks the 06:00 margin is refused here as process, not only as math.

Detailed Core Concepts

What DataOps is, and what it is not

DataOps is the engineering practice of delivering trustworthy data products with the same discipline software teams use for services: versioned artifacts, automated checks, environments, ownership, and reversible change. The product is a table, a topic, a mart, a dashboard contract, or a published grain. The customer is a person or a system that consumes that product on a clock.

DataOps is not:

• A single vendor's observability console renamed.
• "We have Airflow" without owners or promotion rules.
• A central team that must approve every SQL tweak and therefore becomes the bottleneck and the single point of delay.
• A domain free-for-all where every analyst can alter production DDL because "it is just a table."

Without DataOps, the platform looks finished in Module 5 and fails in the first quiet week: a schema drifts, a job succeeds with the wrong definition, a pager goes to the wrong person, and the board is "green" while the number is wrong. The invoice from lesson 58 can still look healthy. Wrong is cheaper than a scan spike until someone notices.

Ownership is the first control

Every certified asset needs three names:

1. Product owner — who decides the grain and the definition_version (product or analytics lead for the board).
2. Engineering owner — who changes the pipeline, the DDL, and the DAG (a named engineer or a small rotation).
3. On-call — who is paged when the publish fails or the freshness cursor stalls (may rotate; must be reachable).

For TechLearningPro:

A RACI that says "Data Team" is not ownership. Ownership is a person or a rotating role with a calendar. Catalog entries without owners are decorations. Lineage without owners is a map of a city with no street names.

Platform team versus domain team

Platform owns shared infrastructure and shared publish mechanics: the lake buckets or catalogs, the warehouse or query engine, the orchestration cluster, the CI templates, the identity patterns from lesson 57, the cost ceilings from lesson 58, the compaction jobs that every domain needs, and the gates that protect production. Platform does not own every business column.

Domain owns the meaning of the data product: the completion rule, the grain of the attempt, the columns the board needs, the tests that prove the grain, and the runbook steps that are product-specific. Domain ships changes through the platform's environments and CI, the way an application team ships through a shared Kubernetes cluster without owning the control plane.

The boundary fails in two classic ways.

Platform as bottleneck. Every new lesson column waits two weeks for a platform ticket. Domains invent shadow pipelines. Definitions drift. The board reads a private table that has no CI.

Domain as free agent. Domains create production tables in the shared catalog, grant broad roles, and schedule DAGs that share the board reservation. Monday fails because a domain backfill took every slot. Lesson 58's isolation rule had no enforcement.

The repair is a contract of interfaces. Platform publishes: environments, CI templates, a publish API or a standard dbt/Spark job pattern, capacity pools, and a change calendar. Domain publishes: models, tests, ownership metadata, and a runbook section. Neither side edits the other's side without review.

Environments are not folders with the same data

A useful environment ladder for this course:

Copying production wholesale into staging "so tests are real" creates a privacy and cost problem. Lesson 61 will classify columns; this lesson refuses to treat a full prod clone as a free CI fixture. Prefer a sampled day of progress with synthetic learner ids, or a masked staging lake that preserves cardinality without emails.

Environment parity means:

• Same partition columns and definition_version semantics.
• Same job identity pattern (service principal / workload identity), different secrets.
• Same DAG structure; different connection ids and schedules (staging may run on demand).
• Same CI checks; prod adds the human change window.

Parity does not mean identical cluster size. Staging can be smaller. It must still catch a missing partition filter and a grain regression.

CI for data is broader than unit tests

Software CI compiles and runs unit tests. Data CI must also answer: will this change publish a wrong grain, drop a required column, introduce a forbidden column, break a downstream contract, or blow the Monday byte budget?

A practical CI ladder for the completion mart:

1. Static. Lint SQL and Python. Forbid SELECT * in certified paths. Fail if definition_version is missing from the publish filter.
2. Contract. Compare schema to the registered contract (Avro/JSON Schema/Protobuf for events; column contract for the mart). Fail on incompatible type change or renamed grain key without a version bump.
3. Grain and quality. Run expectations on a fixture day: uniqueness of attempt key, non-null lesson_id, prohibited columns absent from the published grain.
4. Dry-run cost. Where the engine supports it, dry-run the certified board SQL and fail if scanned bytes exceed a budget (lesson 58).
5. DAG parse and import. Load the DAG; fail on import errors and on schedule collisions with the freeze window.
6. Staging publish. Apply the model to staging; compare row counts and a hash of the grain for the fixture day against a golden file.

A green unit test that never saw a table commit is not enough. A table commit without a version bump when the completion rule changes is a silent board defect.

Change management for pipelines and tables

Treat these as first-class changes:

• DAG code and schedule.
• Spark or dbt transformation that writes the mart.
• Iceberg/Delta schema evolution and partition evolution.
• Warehouse DDL and grants.
• Board SQL and BI extracts pinned to definition_version.
• Kafka topic partition count and retention (with streaming owners).

Each change needs: a ticket or PR, a risk class, a review, a promotion path, and a rollback note.

Risk classes for TechLearningPro:

Rollback for a table is not always git revert. It may be: republish previous snapshot, pin the board to the previous definition_version, or pause the new DAG and leave the last good partition. Write the rollback in the PR before merge. A PR that says "we will figure it out" is not ready.

Runbooks are decision trees, not poetry

A runbook for "board late or wrong" should start with observations, not with capacity.

1. Did the publish DAG succeed for today's event_date?
2. What is max(publish_time) and definition_version on the mart?
3. Does the board SQL filter that version?
4. Did volume fall by more than the expected weekend drop?
5. Is the warehouse or reservation queueing ELT against the board pool?
6. Only then: capacity, resume, or cluster health.

Lesson 58 already refused to resize a warehouse to survive a raw JSON scan. The runbook must refuse the same mistake. "Restart" is last, not first. Ownership tells you who runs which step. Platform runs capacity. Domain runs definition and SQL. Both look at the publish cursor.

Control plane versus data plane

Borrow the language carefully. The control plane is CI, catalogs, IAM patterns, orchestration metadata, and change calendars. The data plane is the bytes, the topics, the queries, and the board. DataOps keeps the control plane from becoming tribal knowledge and keeps the data plane from being edited without the control plane noticing. A notebook that writes production is a data-plane change with no control plane. Ban it for certified paths. Exploration notebooks stay behind the cutoff workgroup from the cost lesson.

Architecture and Design Diagrams

Ownership and promotion sit between authoring and the Monday board.

mermaidCopy
flowchart LR
  DEV["Dev: models and tests"] --> CI["CI: lint contract grain dry-run"]
  CI --> STG["Staging publish"]
  STG --> GATE["Change gate and window"]
  GATE --> PROD["Prod mart completion-v4"]
  PROD --> BOARD["Monday board 06:00"]
  OWN["Owners and on-call"] --> GATE
  OWN --> RUN["Runbook"]
  PROD --> RUN
  BOARD --> RUN

Platform provides the rails; domains ship products on those rails.

mermaidCopy
flowchart TB
  subgraph platform [Platform]
    ENV["Environments and CI templates"]
    CAP["Capacity pools and isolation"]
    CAT["Catalog and identity patterns"]
  end
  subgraph domain [Domain Learning Analytics]
    MODEL["Completion model and grain tests"]
    SQL["Certified board SQL"]
    RB["Product runbook section"]
  end
  ENV --> MODEL
  MODEL --> CAT
  SQL --> CAP
  RB --> CAP

Technical Explanation

Why environment promotion fails when only code is promoted

A common failure: the PR merges SQL that references a column added in a notebook on production last week. Staging never had the column. CI used mocks. Production "works" until the next full rebuild. The repair is to promote schema and fixtures with the code. Staging applies the same migration. The fixture day includes the new column or the test fails. Production never receives a column that only exists because someone typed DDL by hand.

Iceberg and Delta make this sharper. A schema evolution in prod that never ran in staging means the metadata file history diverges. Compaction and expire snapshots behave differently. Promote the table operation as a reviewed job, not as a console click.

How CI should treat definition_version

When the completion rule changes from completion-v3 to completion-v4:

1. Add the new version as a new code path or a new model output; do not silently overwrite the meaning of v3.
2. Update contract tests and golden grain hashes for a fixture day under v4.
3. Keep the board pin on v3 until analytics signs the cutover.
4. Cut over the certified SQL and invalidate BI extracts (lesson 58).
5. Only then deprecate v3 with a date.

CI that only checks "job succeeded" will green-light a silent meaning change. CI that checks the version string and the golden grain will fail the silent change.

Orchestration as a change-controlled surface

Airflow (or Composer, MWAA, Dagster, Prefect—the pattern is the same) stores DAGs as code. Treat connection ids, pool names, and schedules as configuration reviewed with the DAG. A schedule moved from 04:00 to 05:30 can miss 06:00 after a slow merge. That is a high-risk change even if the Python is identical. Put schedule diffs in the PR description. Block merges that move the completion publish into the freeze window without an exception ticket.

Pools and concurrency: domain DAGs must declare the pool that lesson 58 isolated for ELT. CI can grep for a forbidden pool name on certified DAGs. Soft culture fails; a failing check holds.

Catalog as the ownership database

Whether you use an open catalog, a cloud Data Catalog, or a warehouse information schema plus a spreadsheet you hate, the fields that matter for DataOps are: owner, backup, classification (preview of lesson 61), freshness target, upstream jobs, and certified query id. Lesson 22 taught lineage. This lesson makes lineage actionable: an alert without an owner is noise; an owner without a catalog entry is tribal knowledge that leaves when the person leaves.

Change freezes that do not freeze the company

A freeze from Sunday 12:00 to Monday 12:00 for high-risk completion changes protects the board. Low-risk documentation and staging work continue. Emergencies use a break-glass path: two approvers, a written rollback, a post-incident review. A freeze with no break-glass produces shadow changes. A freeze that blocks all SQL produces backlog and then a flood of high-risk merges on Tuesday. Tune the freeze to the product clock, not to a cargo-cult calendar.

Platform engineering metrics that are not vanity

Useful: time from PR open to staging green; percent of prod table changes with a rollback note; percent of certified assets with a named on-call; number of Monday incidents caused by Friday merges; CI flake rate. Vanity: number of DAGs; number of tables; "deployment frequency" that counts notebook saves. Measure what prevents a wrong board, not what looks like DevOps theater.

Branching, releases, and the data trunk

Most teams should keep transformation code on a short-lived branch off a main trunk, the same way application services do. Long-lived "data science" branches that diverge for months create merge conflicts in grain logic that no one can reconcile on a Sunday night. Feature flags for data are awkward; prefer versioned outputs (completion-v4 beside v3) over flags that switch meaning inside one table. Release trains work when many domains share one orchestration cluster: a twice-weekly promotion window for medium-risk changes, with high-risk completion changes still tied to the analytics sign-off and the freeze rules above.

Hotfixes follow the same CI path with a shorter review SLA, not a bypass. A hotfix that skips grain tests to "save the board" is how you publish a second defect while repairing the first. If the board is already wrong, pin to the last good version first, then hotfix forward behind the pin.

Secrets, identities, and who may run what

Lesson 57 chose workload identities and audit scope. DataOps decides which pipeline role may write which database and which human role may assume break-glass. Developers get read on staging and none on prod certified tables by default. On-call gets time-bounded elevation that is logged. CI uses a deploy identity that can apply migrations in staging always and in prod only through the gated workflow. Personal access keys in DAG code are an automatic CI fail. Rotation is a scheduled change with an owner, not a surprise when a key expires at 05:40.

Backfills as first-class changes

A backfill is a production write with a wide blast radius. It needs a ticket, a date range, a pool that is not the board pool, a concurrency limit, and a check that definition_version for historical days matches what consumers expect. Rebuilding two years of completion under v4 rules while the board still pins v3 for recent days is a coherent design only if the SQL and the docs say so. Rebuilding quietly under the same version string with new logic is a historical rewrite that finance and product will treat as fraud even when it was a bug fix. Say what you are rewriting, for which dates, and who approved the semantic change.

Incident communication that matches ownership

When the board is late, the status page or the internal channel should say: which asset, which owner is driving, whether the issue is freshness or correctness, and the next update time. "Platform is investigating" when the definition drifted is how trust dies. Domain drives definition incidents. Platform drives lake, warehouse, and identity incidents. Joint incidents name both. Lesson 60 will add automated signals; this lesson makes sure a human already knows who speaks.

How DataOps absorbs lessons 23 and 58 without duplicating them

Lesson 23 taught pipeline tests and CI/CD mechanics. This lesson decides which of those tests are merge blockers for certified paths and who owns the red build. Lesson 58 taught meters and refusals. This lesson turns dry-run budgets and pool isolation into gates and turns "do not trade the grain check" into a policy a reviewer can enforce. You should not re-derive Spark file-size targets here; you should refuse a PR that disables compaction on the progress landing without a platform owner and a cost note.

Practical Implementation or Design Walkthrough

Problem

TechLearningPro will bump completion from v3 to v4 next month. Domain analytics owns the rule. Platform owns the lakehouse publish path and the warehouse. Design the DataOps path: ownership, environments, CI gates, change window, and the first page of the runbook. Do not invent a proprietary API. Use PR checks, SQL tests, and orchestration pools you already know from earlier lessons.

Ownership record (checked into the repo)

yamlCopy
# ownership/completion_mart.yaml
asset: published.completion_attempt
product_owner: analytics-board
engineering_owner: learning-analytics-de
oncall: learning-analytics-oncall
escalate_to: data-platform-oncall
definition_versions:
  current: completion-v3
  next: completion-v4
  cutover_gate: analytics-signoff
freshness_target: "Monday board ready by 06:00 local"
certified_query: sql/board_completion_attempts.sql
ci_bundle: .github/workflows/completion_mart.yml

CI workflow sketch

yamlCopy
# .github/workflows/completion_mart.yml
name: completion-mart
on:
  pull_request:
    paths:
      - "transforms/completion/**"
      - "sql/board_completion_attempts.sql"
      - "tests/completion/**"
      - "ownership/completion_mart.yaml"
jobs:
  static-and-contracts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Forbid star selects in certified SQL
        run: |
          if grep -E "SELECT\\s+\\*" sql/board_completion_attempts.sql; then
            echo "certified SQL must project columns" >&2
            exit 1
          fi
      - name: Require definition_version predicate
        run: |
          grep -q "definition_version" sql/board_completion_attempts.sql
      - name: Contract and grain tests
        run: |
          pip install -r transforms/completion/requirements.txt
          pytest tests/completion -q

Grain test on a fixture day

pythonCopy
# tests/completion/test_grain.py
import pandas as pd

REQUIRED = {"attempt_id", "learner_id", "lesson_id", "event_date", "definition_version"}
FORBIDDEN = {"email", "raw_payload"}


def test_fixture_day_grain():
    df = pd.read_parquet("tests/completion/fixtures/day_sample.parquet")
    missing = REQUIRED - set(df.columns)
    assert not missing, missing
    leaked = FORBIDDEN & set(df.columns)
    assert not leaked, leaked
    assert df["attempt_id"].is_unique
    assert df["definition_version"].isin(["completion-v3", "completion-v4"]).all()
    assert df["lesson_id"].notna().all()

Staging publish check

sqlCopy
-- Compare staging publish for the fixture date against golden counts.
SELECT
  definition_version,
  COUNT(*) AS attempts,
  COUNT(DISTINCT learner_id) AS learners
FROM staging.completion_attempt
WHERE event_date = DATE '2026-11-03'
GROUP BY definition_version;

Store the expected counts next to the test. A grain change that alters counts without a version bump fails review even if the SQL parses.

Change ticket fields (minimum)

• Risk class (low/medium/high).
• Environments touched.
• Rollback (republish prior version / pin board SQL / pause DAG).
• Capacity impact (pool, expected runtime, collision with 06:00).
• Privacy impact (new PII column? defer details to lesson 61, but the checkbox exists now).
• Owners approving.

Runbook page one

textCopy
Symptom: Monday board late or numbers disputed
1. Check DAG completion_publish for event_date = today (or board date).
2. SELECT max(publish_time), definition_version FROM published.completion_attempt
   WHERE event_date = :board_date;
3. Confirm board SQL pins the expected definition_version.
4. If DAG failed: domain on-call fixes transform; do not resize warehouse first.
5. If DAG succeeded and version wrong: stop cutover; pin previous version; page product owner.
6. If warehouse queue deep: check ELT pool vs board pool (lesson 58 isolation).
7. Escalate to platform on-call only for capacity, identity, or lake outage.

What you will not allow in this walkthrough

• Production DDL from a personal IAM user.
• Merging v4 meaning into the v3 string.
• Skipping staging because "it is only a column."
• A runbook whose first step is reboot.

Cutover checklist for completion-v4

Use this as the PR description template when the version actually ships.

1. Contract file updated; incompatible changes called out.
2. Fixture day golden hashes regenerated under v4 only; v3 fixtures still pass for the legacy path.
3. Staging publish compared for overlapping dates; differences explained in the ticket.
4. Board SQL PR ready but not merged until analytics sign-off.
5. BI extract invalidation steps named with owners.
6. Freeze calendar checked; cutover not scheduled into Sunday–Monday without break-glass.
7. Rollback pin command or SQL snippet pasted in the ticket (not "we will pin").
8. Lesson 60 monitors updated to expect either version during the overlap window, then only v4.
9. Lesson 61 classification reviewed if new columns include email or free-text that might hold PII.
10. Post-cutover review scheduled for the first Monday after go-live.

Sample certified SQL pin

sqlCopy
-- sql/board_completion_attempts.sql
-- Certified for Monday board. CI forbids SELECT *.
SELECT
  lesson_id,
  COUNT(*) AS attempts,
  COUNT(DISTINCT learner_id) AS learners
FROM published.completion_attempt
WHERE event_date BETWEEN DATE '{{ board_start }}' AND DATE '{{ board_end }}'
  AND definition_version = 'completion-v3'  -- bump only with cutover PR
GROUP BY lesson_id;

When v4 goes live, the string changes in the same PR that analytics signs. Two dashboards that hard-code different versions are two products; mark one uncertified.

Orchestration pool declaration (reviewable)

pythonCopy
# transforms/completion/dag_completion_publish.py
# Pseudocode shape for Airflow 2.x-style DAGs; adjust imports to your runtime.
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG(
    dag_id="completion_publish",
    start_date=datetime(2026, 1, 1),
    schedule_interval="0 4 * * *",  # schedule diffs are high-risk
    catchup=False,
    tags=["certified", "completion"],
) as dag:
    publish = PythonOperator(
        task_id="publish_completion_mart",
        python_callable=lambda: None,  # real callable loads the model
        pool="elt_merge",  # CI fails if this becomes board_interactive
    )

A grep in CI for pool="board_interactive" on certified ELT DAGs is a one-line gate that prevents the lesson 58 failure mode from returning as culture.

Enterprise Real-World Scenarios

TechLearningPro: Friday merge, Monday blame

A domain engineer adds time_spent_seconds to the mart on Friday afternoon. CI has unit tests but no staging publish and no certified SQL dry-run. The board SQL still selects an explicit list and ignores the new column, so the board looks fine. A second dashboard, built by a contractor, uses SELECT * into a BI extract. Monday's extract widens, a join duplicates rows, and completion rates drop ten points. Platform is paged. The warehouse is healthy. The extract is wrong.

DataOps repair: forbid star selects in anything labeled certified or extract-feeding; require staging publish; invalidate extracts on schema change; add the column as medium risk with analytics review; document that extracts subscribe to a column contract, not to *. Ownership of the contractor dashboard moves under analytics or the dashboard is unmarked as certified.

Northline: platform bottleneck births a shadow mart

Northline finance waits three weeks for a refund status column. A finance analyst loads a CSV into a personal schema nightly. The portal still uses the certified table. Month-end uses the CSV. Refund logic drifts by one status code—the same failure mode lesson 58 used for a second warehouse. Here the second mart is a person, not an engine.

DataOps repair: platform publishes a self-serve path for additive optional columns with contract tests and automatic staging. SLA for medium-risk columns is measured in days, not weeks. Shadow tables are detected by query history against non-certified schemas and either promoted through CI or revoked. Platform stops being the typist for every column and starts being the gate for production meaning.

A streaming domain changes retention without telling the lake

The Kafka progress topic retention is cut to save disk. The batch catch-up job that rebuilds a bad day can no longer read two days back. Compaction still works for landed Parquet, but a mid-flight outage loses the replay window. Platform on-call follows a runbook that says "increase consumers." Consumers are fine. The retention change was a high-risk streaming change with no cross-team review.

DataOps repair: topic retention and partition changes require streaming + lakehouse owners on the PR. The ownership YAML for the topic lists the replay consumers. CI does not catch Kafka cluster config if it lives outside git—so cluster config moves into the same change process or a documented exception with a calendar review.

Healthcare-adjacent learning records and the change calendar

A TechLearningPro enterprise customer in a regulated industry treats completion evidence as part of compliance training attestation. Their contract requires that historical completion rows not silently change meaning after the fact. The same DataOps rules apply with sharper teeth: backfills that alter attestation dates need legal or compliance co-approval; definition_version cutovers need a customer communication plan; freeze windows expand around audit season. The engineering controls are identical—ownership, CI, version pins—but the change calendar gains stakeholders who are not engineers. Ignoring them produces a correct warehouse and a broken contract.

Multi-cloud reopen conditions under change control

Lessons 54 through 56 left reopen conditions for a second engine. DataOps forbids a nightly job on the loser of the bake-off, but it also forbids a quiet weekday experiment that loads production completion-v4 into a second warehouse "just to see." Experiments use staging data or a time-boxed project with a teardown date in the ticket. When a reopen condition is exercised for real, it is a high-risk change: dual publish, dual ownership, dual runbooks, and an explicit end state (fail back or cut over). Without that, the cost sheet from lesson 58 grows a second H and the definition drifts again.

Best Practices

• Name a product owner, engineering owner, and on-call for every certified asset; store them next to the code.
• Separate platform rails from domain products; measure platform lead time so domains do not build shadows.
• Promote schema, fixtures, and schedules with code; do not promote only Python files.
• Put grain, forbidden-column, contract, and dry-run byte checks in CI for certified paths.
• Version meaning with definition_version; never silently redefine a version string.
• Classify change risk and protect the Sunday–Monday publish window with a freeze that still has break-glass.
• Write rollback before merge; table rollback is often pin-and-republish, not git revert alone.
• Start runbooks at publish cursor and definition, not at cluster restart.
• Keep staging masked or sampled; full prod clones are a privacy and cost incident waiting for lesson 61.
• Enforce pool isolation in CI so domain backfills cannot steal the board reservation.
• Track CI flake and time-to-staging-green; ignore vanity DAG counts.
• Hand observability (lesson 60) a list of owned assets and expected freshness before you add alerts.

Common Mistakes and Anti-Patterns

Ownership = "the data team." Nobody is paged. Everyone is blamed. Write names and rotations.

CI that only unit-tests pure functions. The bug was in the table grain and the extract. Add contract, grain, and staging publish checks.

Staging that is a full production clone. You duplicated PII and the bill. Sample and mask; keep schema parity.

Platform must approve every column. Domains invent CSV marts. Publish a self-serve additive path with automated gates.

Domains edit production DDL in the console. History diverges; CI is theater. Ban console DDL for certified tables.

Runbook step one: restart the warehouse. Lesson 58 already showed the cost of that reflex. Diagnose publish and definition first.

Silent reuse of completion-v3 for a new rule. The board cannot tell truth from nostalgia. Bump the version and pin the cutover.

Freeze with no break-glass. People bypass the freeze in chat. Document emergency approval or the freeze is fiction.

Kafka or warehouse config outside the change process. Git is green while production retention moved. Bring config into review or schedule audits.

Measuring success as number of deployments. A flood of unowned tables is not DataOps. Measure owned certified assets and Monday defect rate.

Troubleshooting and Design Review

Monday board wrong, all DAGs green. Check definition_version on the mart versus the board SQL and any BI extract. Green jobs can publish the wrong meaning. Diff the extract schema against the contract.

CI always red on flake. Quarantine flaky tests with an owner and a burn-down; do not skip the grain check to go green. Flake that trains people to ignore CI is an outage generator.

Staging passes, production fails. Secrets, pool names, data volume, or a partition that only exists in prod. Compare connection ids and run the same dry-run bytes in prod's measurement workgroup with a cutoff.

Domain says platform is too slow. Publish lead-time metrics and the self-serve column path. If medium-risk changes still take weeks, the bottleneck is real; fix the path before blaming culture.

On-call pages platform for every dashboard complaint. Route by ownership YAML: definition and SQL to domain; capacity and lake to platform. Misrouted pages train platform to guess at business rules.

A change rolled back in git but the table still serves new data. Git revert does not rewind Iceberg/Delta snapshots or warehouse commits. Execute the rollback note: pin version, republish prior snapshot, pause the new writer.

Design review checklist:

• Does every certified asset have three names (product, engineering, on-call)?
• Do CI gates cover contract, grain, forbidden columns, and dry-run budget?
• Is staging schema-parity without being a full PII clone?
• Is there a written rollback for the next definition_version cutover?
• Does the runbook start with publish cursor and version?
• Are ELT pools enforced so a backfill cannot starve 06:00?
• What is the measured time from PR to staging green for a medium-risk column?

Interview Questions and Answers

Q1. What is DataOps in one sentence that is not a vendor pitch?

DataOps is operating data products with ownership, automated checks, environments, and reversible change so a table or pipeline change is as reviewable as a service change. Tools implement pieces of it. Ownership and gates are the practice.

Q2. How do you split platform and domain responsibilities for a completion mart?

Platform owns environments, CI templates, capacity isolation, catalog patterns, and the shared publish mechanics. Domain owns the completion grain, definition_version meaning, certified SQL, product tests, and the product-specific runbook. Domain ships through platform rails. Neither silently edits production outside that path.

Q3. Why is a green unit test insufficient for a mart change?

Unit tests rarely see schema evolution, partition predicates, extract consumers, or byte budgets. I require contract tests, grain and forbidden-column checks on fixtures, a staging publish, and a dry-run of the certified query when the engine allows it. Meaning changes need a version bump, not only a green pytest.

Q4. How do you roll back a bad table publish?

I follow the rollback written in the PR: pin consumers to the previous definition_version, republish or restore the prior snapshot or partition, and pause the new writer. I do not assume git revert undoes an Iceberg commit or a warehouse load. I invalidate BI extracts that cached the bad schema.

Q5. What belongs in a Monday-board runbook's first five steps?

Confirm DAG success for the board date; read max(publish_time) and definition_version; confirm the board SQL pin; check volume against expected seasonality; check pool isolation if the warehouse is busy. Capacity restart comes after those, not before.

Q6. When should a domain be allowed to add a column without a platform engineer typing DDL?

When the column is additive, optional, covered by contract tests, classified for privacy, and promoted through staging with an ownership update. Platform provides the path and the gates. Breaking grain changes and partition evolution stay dual-approved.

Q7. How do you keep staging useful without copying all production PII?

Keep schema and partition semantics identical; use sampled days and masked or synthetic identifiers; run the same CI checks. Full clones create privacy risk and cost. Cardinality can be preserved without emails and phone numbers.

Q8. Why treat Airflow schedule changes as high risk even when code is unchanged?

Because the publish margin before 06:00 is part of the product. Moving a schedule later can miss the board with identical transformation logic. I require schedule diffs in the PR and block freeze-window collisions.

Q9. How would you detect a shadow mart like Northline's CSV?

Query history and warehouse access logs for heavy reads on non-certified schemas; catalog drift reports; month-end reconciliation between certified and ad-hoc sources. Then either promote the logic through CI or revoke the grants that make the shadow easy.

Q10. What metrics prove DataOps is working?

Named ownership coverage for certified assets; time to staging green; percent of prod changes with rollback notes; Monday defects from unreviewed changes; CI flake rate. Not raw DAG count or "we deployed a lot."

Q11. How does this lesson use the cost sheet from lesson 58?

Dry-run byte budgets become CI gates; pool isolation becomes an enforceable check; refusals (no raw board scans, no dropped grain checks) become merge blockers. DataOps is how the cost sheet stops being a slide and becomes a control.

Q12. What do you hand to the observability lesson next?

An owned asset list with freshness targets and definition versions, so alerts page the right person and monitor the right cursor. Alerts without owners are noise; owners without monitors are hope.

Hands-On Exercises

Exercise: DataOps operating model for the completion board

Problem statement. Define a DataOps operating model for TechLearningPro's published.completion_attempt mart and its Monday board. Include ownership, environment promotion, CI gates, a change window, and runbook page one for a late or wrong board. Assume completion-v4 will replace completion-v3 within a month. Do not invent proprietary control-plane APIs; use YAML ownership, PR checks, SQL/Python tests, and orchestration pools.

Tasks.

1. Write an ownership record for the mart, the certified SQL, and the publish DAG (product, engineering, on-call, escalate).
2. Describe dev, staging, and prod data rules (what is sampled or masked; what must stay identical).
3. List CI checks that must fail the v4 cutover PR if grain or forbidden columns regress.
4. Classify the v4 cutover risk and write the rollback steps (pin, republish, pause).
5. Draft runbook page one that does not start with restarting capacity.
6. State how platform vs domain split responsibilities for an additive column versus a grain change.
7. Note one metric you will track for the next quarter to prove the model works.
8. Document limitations (what CI cannot catch; what still needs a human change window).

Expected deliverables. Ownership YAML (or equivalent table), environment rules, CI checklist, cutover/rollback plan, runbook page one, platform/domain RACI for two change types, one success metric.

Evaluation rubric.

Working with contractors and temporary staff

Contractors who build dashboards need a time-bounded identity, a staging-only default, and a certified-query template. Their extracts expire. Their ownership YAML entries have an end date. When the contract ends, access reviews from lesson 61 and IAM from lesson 57 remove them; DataOps removes their DAGs from the certified tag if nobody renews ownership. Orphaned contractor pipelines are a common Monday surprise.

Documentation that survives the person

The minimum living docs next to code: ownership YAML, contract files, runbook page one, cutover checklist, and a one-page architecture link diagram for the certified chain. Long Confluence novels rot. If the doc is required for a high-risk change, store it in the repo and review it in the PR. If nobody read it in ninety days, delete or rewrite—it is worse than nothing when it lies.

Game days for change failure

Once a quarter, practice: pin the board to the previous definition_version, pause the publish DAG, and restore a staging publish from the prior artifact. Time the steps. Update the rollback notes with real durations. Lesson 58 wanted restore drills for replicas; DataOps wants restore drills for meaning. A rollback that only exists as prose will fail at 05:40.

Definition of done for a certified data product

A product is not done when the SQL returns rows in a notebook. It is done when: ownership YAML exists; contracts and grain tests live in CI; staging publish is exercised; prod promotion used the change gate; monitors from lesson 60 are registered; classification from lesson 61 is set; runbook page one is linked; cost dry-run budget is recorded; and the board SQL pins definition_version. Anything missing is explicit debt with an owner—not a vague backlog wish.

How to refuse hero culture politely

When someone asks to "just quickly update production," answer with the risk class and the next gate. Offer to pair on a same-day hotfix through CI if the board is already broken. Hero pushes without evidence create the next incident. DataOps is slower than a console click and faster than a week of forensic SQL.

Interface contracts between platform and domain (checklist)

Platform provides: environment templates, CI reusable workflows, pool names, catalog schema for ownership fields, secret patterns, observability sinks, and a change calendar API or doc. Domain provides: models, tests, ownership values, runbook sections, monitor threshold proposals, and classification drafts. Pull requests that change platform templates require platform review. Pull requests that change grain require domain product sign-off. Dual review for high-risk cutovers. Write this once; link it from every onboarding doc.

Lesson Summary

DataOps is how Module 6 keeps the Module 5 platform from rotting under unowned change. Name owners. Separate platform rails from domain products. Promote schema and fixtures with code. Put grain, contract, and cost dry-runs in CI. Version meaning explicitly. Freeze the publish window without forbidding emergencies. Write runbooks that read the publish cursor before they touch capacity. The Monday board fails as process long before it fails as Spark.

Data Observability and Operational Monitoring attaches monitors and pages to the assets you just named. Data Governance, Privacy and Compliance classifies what those assets may contain and who may see them. Data Reliability and SLA/SLO Design will turn 06:00 into an SLO; arrive with a change gate that already respects that clock.

Further Reading

• DataOps Manifesto for principles; treat vendor claims separately from the ownership and CI practices in this lesson.
• Apache Airflow documentation for DAG-as-code, pools, and deployment patterns that belong under change control.
• dbt documentation for model CI, contracts, and environment promotion patterns when SQL models are your publish path.
• Great Expectations or similar expectation libraries for grain and column checks in CI (concepts transfer even if you use another library).
• Apache Iceberg docs and Delta Lake docs for schema evolution and snapshot rollback as operational controls.
• Lessons 23, 57, and 58 for CI test shapes, identity, and cost gates this operating model enforces.
• Lesson 60 and 61 briefs in this module for the monitors and privacy controls that hang on ownership.