Data Contracts and Schema Governance

Data Engineering & Modern Data Platforms · Lesson 69 · Production Data Engineering and Capstone

Lesson Overview

The transform was green. The gate was green. The Monday board still lied. Learning had added device_class to progress as a required string. Billing's enrollment export renamed plan_code to sku_code in the warehouse table the completion job still joined. Identity shortened learner_key documentation and began emitting a new opaque id in a parallel column without telling the mart owner. None of these changes failed a Spark job on the first day. They failed meaning. Compatibility at the byte level is not the same as a promise between teams.

Schema Registry and Event Contracts taught you subjects, schema ids, and backward versus forward rules on a Kafka topic. This lesson does not clone that Kafka course. It lifts the same discipline to the estate: producer-consumer contracts for events, tables, and APIs; ownership; evolution rules; breaking versus compatible changes; contract tests in CI; and communication when a break is unavoidable. TechLearningPro's progress→completion platform from Enterprise Data Platform Capstone Project only stays certified if Learning, the platform team, and every warehouse consumer share a written contract—not a Slack thread that scrolled away.

A contract is the interface. A schema is necessary structure inside that interface. Governance is how versions are proposed, tested, accepted, and retired. Without ownership, a registry is a junk drawer. Without CI tests, a wiki is theater. Without semantic rules, a compatible add still ships the wrong money unit.

Learning Objectives

By the end of this lesson you will be able to:

1. Define a data contract as a producer-consumer agreement covering schema, grain, semantics, SLOs, and ownership—not only a JSON Schema file.
2. Separate event contracts, table contracts, and API contracts, and say which TechLearningPro surfaces use which.
3. Connect registry compatibility levels from lesson 43 to warehouse and lakehouse evolution without re-teaching Kafka serializers.
4. Classify changes as compatible or breaking for readers and writers, including renames, type changes, nullability, and semantic shifts with unchanged types.
5. Write ownership: who may propose, who must approve, who upgrades first, who is paged when the contract fails.
6. Put contract tests in CI: schema compatibility, example payloads, consumer expectations, and mart column gates.
7. Version contracts alongside definition_version for metrics that are not structural fields.
8. Design a breaking-change playbook: dual publish, dual read, cutover, retirement.
9. Apply contracts to Learning progress, Billing exports, and Identity keys with explicit cross-domain joins.
10. Explain what contracts do not catch, and which quality and privacy controls still sit beside them.

Prerequisites

• Lesson 43 for registry subjects, compatibility levels, defaults, and the limits of structural checks.
• Data Quality Engineering and Data Pipeline Testing and CI/CD for gates and pipeline tests.
• Data Lineage and Metadata Management for discovering consumers before you break them.
• Module 5 lakehouse and mart lessons so table contracts have a physical home (Delta/Iceberg/warehouse).
• Lesson 68's capstone charter: you should already have at least one progress contract artifact to harden here.
• Lesson 61 for prohibited fields and privacy constraints that belong in the contract's negative space.

Detailed Core Concepts

What a data contract is

A data contract is a versioned agreement between a producer and one or more consumers about a dataset or event stream. It typically includes:

• Identity: name, owners, domains, environments (dev/stage/prod).
• Schema: fields, types, nullability, enumerations, nested shapes.
• Keys and grain: what one row or one event means; primary and business keys.
• Semantics: units, timezones, status meanings, definition_version ties for derived metrics.
• Compatibility policy: backward, forward, full, transitive; or warehouse-specific rules.
• Quality promises: freshness SLO, completeness expectations, known limitations.
• Interfaces: topic + subject, table + schema name, or HTTP/RPC endpoint + media type.
• Change process: how to propose, test, announce, dual-run, and retire.
• Negative space: prohibited fields, forbidden joins, fields that must never appear in a mart.

A schema file alone is a fragment. "The column exists" does not mean "completion means the same thing." TechLearningPro's definition_version = completion-v4 is a semantic contract even when the physical columns look unchanged from v3.

Three surfaces that need contracts

Event contracts. Progress on learning.progress, purchase lines, CDC envelopes. Lesson 43's registry is the structural enforcer for Avro/JSON Schema/Protobuf subjects. The contract document adds key, event-time field, idempotency key, and who upgrades first.

Table contracts. Lakehouse and warehouse tables such as curated.progress_event, published.completion_attempt, Billing's billing.enrollment_daily. Enforcement is migration review, schema-as-code, column-level tests, and engine features (Delta column mapping discipline, Iceberg schema evolution rules, BigQuery required fields, etc.). Consumers are SQL jobs and BI extracts, not only Kafka clients.

API contracts. Metrics APIs, enrollment exports, identity resolution endpoints. OpenAPI or similar plus consumer-driven contract tests. The completion board may never call an HTTP API, but Identity's key-resolution API is how Billing and Learning avoid inventing a third learner id.

Treat all three as first-class. Teams that only govern Kafka and leave warehouse tables as "whatever dbt compiled" recreate the Monday lie on a slower schedule.

Compatibility, restated for estate governance

Lesson 43's rules still hold for events:

• Backward: new consumer reads old data; upgrade consumers first; add fields with defaults; removals ok for new readers.
• Forward: old consumer reads new data; upgrade producers first; careful removals.
• Full / transitive: both directions; check against history, not only the latest neighbor.

For tables, map the spirit without pretending every warehouse speaks Avro resolution:

• Adding a nullable column is usually compatible for existing SELECT lists that name columns.
• Adding a required column without a default breaks writers and often breaks SELECT * assumptions in poorly written consumers.
• Dropping or renaming a column breaks consumers that name it; prefer add-new + dual-write + deprecate-old.
• Widening types (int→bigint) is often compatible; narrowing is not.
• Changing the meaning of status = 3 without a version bump is a semantic break even if the type is still int.

For APIs, additive optional fields are usually compatible; removing fields, changing status codes, or altering pagination tokens are breaks unless versioned in the path or media type.

Breaking versus compatible: a working catalog

If the catalog and the CI disagree, CI wins until the catalog is fixed.

Ownership

Every contract has:

• Producer owner: the team that writes the bytes or rows (Learning for progress).
• Contract owner: often the producer, sometimes a platform steward for shared dimensions.
• Consumer representatives: at least one for each critical path (completion mart, Monday board, Billing join).
• Approvers for breaking changes: producer + named consumers + privacy if fields touch PII.

Platform teams provide registry, CI templates, and catalog hosting. They do not silently own Learning's meaning of lesson_id. Federated ownership with centralized tooling is the model lesson 70 will push further; here the minimum is "no orphan interfaces."

Contract tests in CI

Tests that belong in the producer pipeline before merge:

1. Schema compatibility against the registry or against the previous committed schema artifact (BACKWARD_TRANSITIVE or your table policy).
2. Golden examples: fixtures that must serialize/deserialize.
3. Prohibited field scan: fail if email appears in the contract or in example payloads destined for the mart path.
4. Consumer stubs: for critical consumers, a pack of expectations (column exists, type, grain check query) runs in CI against synthetic tables or contract mocks.
5. Semver / version resource: if definition_version changes, a checklist job requires board SQL and runbook updates in the same PR.

Consumer pipelines should fail closed when the producer's declared version is newer than the consumer supports, rather than guessing.

What contracts do not catch

• Skewed partitions and cost blowups (lesson 58).
• Late data outside the freshness SLO (lesson 62).
• Correct joins to the wrong grain.
• Privacy lawful basis (lesson 61) beyond "field forbidden."
• Duplicate business meaning under two column names both "compatible."

Quality gates, lineage reviews, and privacy reviews sit beside contracts. Contracts reduce surprise; they do not replace operations.

Architecture and Design Diagrams

Contract flow from proposal to serve. CI is on the producer path; consumers pin versions.

mermaidCopy
flowchart TD
  PROP["Producer proposes schema/contract PR"]
  CI["CI: compatibility + fixtures + prohibited scan"]
  REG["Registry / schema repo / catalog"]
  PROD["Producer writes events or rows"]
  CONS["Consumers pin contract version"]
  GATE["Runtime quality gate"]
  MART["Certified mart / API"]
  BRK["Breaking change playbook"]
  PROP --> CI
  CI -->|pass| REG
  CI -->|fail| PROP
  REG --> PROD --> GATE --> MART
  REG --> CONS
  CONS --> MART
  CI -->|breaking flagged| BRK
  BRK --> REG

TechLearningPro cross-domain contracts without a single god-schema.

mermaidCopy
flowchart LR
  subgraph learning [Learning domain]
    LP["progress event contract"]
    CM["completion mart contract"]
  end
  subgraph identity [Identity domain]
    IK["learner_key resolution API contract"]
  end
  subgraph billing [Billing domain]
    EN["enrollment_daily table contract"]
  end
  LP --> CM
  IK --> CM
  EN --> CM
  IK --> EN

Technical Explanation

Event path: reuse lesson 43, extend the document

For learning.progress-value (or your subject name):

• Keep JSON Schema or Avro in the registry with a transitive compatibility mode appropriate to multiple consumers.
• Document in the contract markdown or metadata: key = learner_key or event_id strategy you chose; event-time = event_time; idempotency = event_id; prohibited payload fields.
• Spark jobs that still parse JSON without calling the registry must pin a schema artifact in the repo and test compatibility offline—the same rule lesson 43 gave for learning.progress staying JSON until migration.

Do not spend this lesson re-explaining magic bytes. Point back to 43 and spend the words on estate process.

Table path: schema-as-code

Store table contracts next to the jobs that publish them:

textCopy
contracts/
  events/learning_progress.yaml
  tables/published_completion_attempt.yaml
  tables/billing_enrollment_daily.yaml
  apis/identity_learner_resolve.yaml

Example table contract (illustrative YAML):

yamlCopy
name: published.completion_attempt
version: 4
owners:
  producer: learning-data
  consumers: [ops-board, learning-science]
grain: [learner_key, lesson_id, attempt_number]
compatibility: backward_additive
fields:
  - { name: learner_key, type: string, required: true }
  - { name: lesson_id, type: string, required: true }
  - { name: attempt_number, type: int, required: true }
  - { name: event_date, type: date, required: true }
  - { name: definition_version, type: string, required: true, enum: [completion-v4] }
  - { name: completed_at, type: timestamp, required: true }
prohibited_fields: [email, payment_token, card_number]
sla:
  fresh_by: "06:00"
  timezone: "America/Chicago"
evolution:
  rename_policy: add_and_deprecate
  breaking_requires: [consumer_ack, dual_run_plan]

CI loads the previous version from git and fails the PR if a rename slipped in without the dual-run plan file.

API path: consumer-driven checks

Identity exposes GET /v1/learners/{alias} returning { "learner_key": "...", "key_version": 2 }. Billing and Learning run contract tests that assert learner_key presence and reject payloads with email. When Identity wants learner_key_v3, it adds a field or a /v2 route; it does not silently change /v1 semantics.

Semantic versions versus schema versions

completion-v4 may share physical columns with completion-v3 while changing checkpoint rules. That change must bump the contract's semantic version and the board's filter in one coordinated release. Structural CI may stay green. Add an explicit test: board SQL contains definition_version = 'completion-v4' when contract semantic version is 4.

Breaking-change playbook

1. Propose PR with breaking: true, affected consumers from lineage, and dual-run plan.
2. Deploy producer dual-write (old + new field/table/topic) or dual-publish.
3. Upgrade consumers to read new; verify parity queries.
4. Stop writing old; keep old readable until retention expires.
5. Remove old field/table/topic in a later change window.
6. Update runbooks and capstone cost/SLO docs if freshness changed.

Skipping to step 5 is how finance and the board diverge for a month.

Enforcement points

Missing any one layer recreates "works in prod until a new consumer appears."

Contract lifecycle states

Treat contracts as versioned products with states:

1. Draft — editable, not consumable in prod.
2. Candidate — CI green in a staging subject/schema; consumers invited to run expectation packs.
3. Active — producers write it; certified consumers may pin it.
4. Deprecated — dual-run or read-only old interface; sunset date required.
5. Retired — removed from produce paths; historical data may still deserialize under archived schemas.

A contract that jumps from Draft to Active on Friday afternoon without Candidate soak is how the board inherits an experiment. The soak window can be short (days) if consumer packs are strong; it cannot be zero for certified marts.

Mapping registry modes to warehouse policies

When a team says "we're cloud-native, we don't need Avro rules," translate: they still need an analogue. Git history of contract YAML plus CI is that analogue for tables.

Worked example: adding device_class correctly

1. Add to JSON Schema / Avro as optional string with default "unknown" (or nullable without default if table policy allows nulls and consumers tolerate them).
2. Bump contract minor version; changelog entry.
3. CI compatibility passes; old fixture without the field still deserializes.
4. Producer mobile app ships gradually; lake job stores null/default.
5. Science dashboard must use coalesce(device_class, 'unknown') until backfill completeness SLO is met.
6. Only then consider tightening nullability in a major revision after backfill.

Incorrect path: required field, registry NONE, dashboard inner-filters to mobile, silent drop.

Worked example: rejecting lessonId camelCase rename

A well-meaning cleanup PR renames lesson_id to lessonId for "JSON style." Compatibility CI shows removal of lesson_id. Review fails. Correct approach: add lessonId only if an external partner requires it, dual-publish, migrate, deprecate lesson_id after the horizon—or reject the rename entirely and keep snake_case as the estate standard. Federated governance (lesson 70) should pick one naming standard so every domain does not bike-shed renames.

Contract tests beyond schema diff

Structural diffs catch renames and type breaks. Add:

• Grain tests: count(*) = count(distinct key tuple) on fixtures.
• Foreign key smoke: every lesson_id in fixture progress exists in manifest fixture or is intentionally unknown under cap.
• Invariant tests: completed_at >= event_time for completion rows.
• Privacy tests: regex/classifier fails on email-like columns in contract field names and example payloads.
• Board pin test: certified SQL file contains the active definition_version literal.

These are still contract tests: they protect the promise, not only the Avro tree.

Producer CI pipeline sketch

textCopy
PR opened
  -> lint contract YAML
  -> compatibility vs last Active tag
  -> prohibited field scan
  -> golden encode/decode
  -> consumer expectation pack (board + billing stub)
  -> if breaking:flag require dual_run_plan.md and approvers
  -> publish Candidate artifact
merge to main
  -> tag Active for env promotion
deploy producer
  -> runtime registry/API validation
  -> publish gate still runs (defense in depth)

Runtime gates remain. CI reduces the rate of bad deploys; it does not replace quarantine for dirty data that is schema-valid.

Consumer pinning strategies

• Exact version pin: board requires contract completion_attempt v4. Safe; requires coordinated upgrades.
• Range pin: accept ^4 compatible minors. Faster; needs trust in semver discipline.
• Always latest: forbidden for certified paths. Acceptable for explorers in sandbox.

Identity API consumers should pin major versions in the URL (/v1) and treat /v2 as a new contract.

Cross-domain contract matrix for TechLearningPro

This matrix belongs in the catalog home page. Lesson 70 will ask who funds ownership; this lesson asks that the matrix exists.

Communication templates

Compatible change:

Subject: [contract][compatible] learning.progress v12 adds device_class

Body: change summary; default value; consumer action (none required / optional adopt); CI link; rollback (revert producer).

Breaking change:

Subject: [contract][breaking] billing.enrollment_daily plan_code → sku_code

Body: dual-run start/end; parity query; consumer owners tagged; upgrade order; risk if missed; privacy checklist N/A or attached; federation approval link.

Without templates, people write novels or nothing.

Relationship to quality SLOs

A contract may promise freshness ("progress land within 15 minutes of event P95") separately from the board's 06:00 mart SLO. Do not overload one number. Landing can be healthy while the mart is late (transform failure). Mart can be on time while landing freshness burns (luck of batch window). Observability must split the signals—lesson 60's hop naming applies inside contract operations too.

Practical Implementation or Design Walkthrough

Problem

TechLearningPro must version a contract for progress events and for published.completion_attempt, prove a compatible add, reject a breaking rename in CI, and define backward-compatibility tests Billing can run before it changes plan_code. Extend—not replace—the lesson 43 registry story.

Step 1 — Write the progress event contract

Include schema reference (JSON Schema file or Avro), key, event-time, idempotency, owners, compatibility mode, prohibited fields, and freshness expectation for landing (not the board SLO).

Step 2 — Write the mart table contract

Grain, fields, definition_version enum, fresh_by 06:00, consumers list including the board, rename policy.

Step 3 — CI compatibility check (sketch)

pythonCopy
"""Illustrative: compare previous and candidate field sets.
Replace with registry API or dedicated contract CLI in real pipelines.
"""
from __future__ import annotations

PREV = {"event_id", "learner_key", "lesson_id", "event_time", "event_date"}
CAND_OK = PREV | {"device_class"}  # optional add
CAND_BAD = (PREV - {"lesson_id"}) | {"lessonId"}  # rename


def assert_additive_backward(prev: set[str], cand: set[str]) -> None:
    removed = prev - cand
    if removed:
        raise AssertionError(f"breaking removal/rename: {sorted(removed)}")


assert_additive_backward(PREV, CAND_OK)
try:
    assert_additive_backward(PREV, CAND_BAD)
except AssertionError as e:
    print("rejected:", e)

Step 4 — Consumer expectation pack for the board

sqlCopy
-- Runs in CI against a synthetic mart or contract-shadow tables
SELECT
  COUNT(*) = 0 AS no_null_keys
FROM published.completion_attempt
WHERE learner_key IS NULL OR lesson_id IS NULL;

SELECT
  COUNT(*) = 0 AS version_pinned
FROM published.completion_attempt
WHERE definition_version <> 'completion-v4';

Step 5 — Billing rename rehearsal

Billing wants sku_code instead of plan_code. Contract PR adds sku_code, dual-writes, keeps plan_code until Learning's join job and any finance extracts migrate. CI fails a PR that drops plan_code without dual_run_plan.md. Parity query: counts by both columns agree during dual-run.

Step 6 — Communication

Change announcement template: contract name, version, compatible/breaking, upgrade order, dual-run window dates, rollback, privacy impact. Post to the consumer list from the contract, not to a random channel.

Acceptance criteria for the walkthrough

• Compatible add merges green.
• Rename without playbook fails CI.
• Board expectations fail if definition_version drifts.
• Billing dual-run plan reviewed by Learning consumer owner.
• Catalog or README links each physical table/topic to a contract version.

Governance for schema registries at estate scale

Once you have more than a handful of subjects, registry operations need policy:

• Default compatibility mode for new subjects (recommend BACKWARD_TRANSITIVE for business events).
• Naming standards for subjects (domain.entity-value).
• Who can call PUT compatibility or delete subjects (break-glass only).
• Soft-delete and recovery procedures.
• Separate registries or contexts for prod vs non-prod so lab NONE modes never touch prod.

Platform owns the registry service. Domains own subject contents. Lesson 43 remains the mechanics manual; this estate policy is why Monday's purchase-line incident does not repeat across fifty topics.

Warehouse migration patterns catalog

Expand-contract. Add new column; backfill; switch readers; drop old. Default for renames.

Parallel table. Create enrollment_daily_v2 with new grain; dual publish; cutover consumers; retire v1. Default for grain breaks.

View shim. Keep table columns stable; expose a view with friendly names. Useful for BI; dangerous if the view hides semantic breaks—pin definition_version in the view SQL.

Blue/green dataset. Two physical datasets behind a pointer, same idea as the capstone publish pointer. Works well with lakehouse branches when available.

Pick a pattern in the dual-run plan; do not invent a fifth on the incident call.

Consumer expectation pack layout

textCopy
consumers/
  board/
    expectations.sql
    contract_pin.txt   # completion_attempt@4
  billing_join/
    expectations.sql
    fixtures/

Producer CI checks out or packages these packs as submodules or published artifacts. When Billing owns a pack against Identity, Billing CI also runs it on Identity staging—consumer-driven testing in both directions as needed.

Change calendar and freeze windows

Certified board metrics should respect freeze windows before major Monday reviews or accreditation exports: compatible changes only, or emergency break-glass with executive + privacy approval. Contracts make freezes enforceable: CI can deny breaking flags during freeze tags.

Security review hooks

Any contract field classified as PII or sensitive triggers lesson 61 workflow: lawful basis, retention, encryption, access. The contract PR template includes a privacy checkbox. Missing checkbox fails CI for fields matching a sensitive name list, with an override requiring privacy steward approval token.

Extended troubleshooting

Consumers pin different majors of the same product. Catalog shows v3 and v4 both Active without deprecation. Fix: only one Active major for certified ports; previous major Deprecated with date.

Registry compatible, Spark job fails. Spark schema literal out of date; job does not fetch registry. Sync artifact or enable documented registry integration.

Expectation pack flaky. Pack queries prod during CI. Point packs at synthetic fixtures or ephemeral schemas loaded in CI.

Dual-run never ends. No owner for cutover date. Federation sets a maximum dual-run duration for certified products.

Enterprise Real-World Scenarios

TechLearningPro: device_class and the quiet dashboard

Learning added device_class as required without a default. Old mobile clients omitted it. The registry was on NONE in a "temporary" subject. The lake job filled nulls. The science dashboard filtered device_class = 'mobile' and quietly dropped half of yesterday. Structural monitoring said volume was fine.

Repair: transitive backward compatibility; required fields need defaults or a new major contract; consumers that filter new dimensions must treat null as "unknown" until backfill completes; contract test includes an old-client fixture.

Northline: warehouse-only governance

Northline governed Kafka tightly after lesson-43-style incidents, then let analysts alter warehouse tables in the console. A "cleanup" dropped a unused-looking column referenced by a monthly compliance extract. Kafka stayed healthy. The extract failed on day 28.

Repair: table contracts in CI for any table marked certified or regulatory; console DDL locked down; extracts pin contract versions; lineage job lists consumers before drop.

Contoso: API contract theater

Identity published OpenAPI and never ran consumer tests. Billing cached learner_key from an undocumented field that disappeared in a "non-breaking" patch. Completions orphaned for two days.

Repair: consumer-driven contract tests in Billing CI against Identity's staged swagger; versioned /v1 frozen; additive fields only; key rotation uses a new field with overlap window.

When a break is the honest choice

completion-v5 changes grain from attempt to lesson-day. That is not an additive column. Ship a new table contract published.completion_lesson_day, dual publish, move the board on a named date, retire v4 after the horizon. Pretending it is compatible is how two boards disagree politely forever.

Schema governance board versus federation

Some enterprises create a "schema review board" that meets weekly and approves every field. That model collapses under change volume and becomes rubber-stamp theater. Prefer:

• Automated compatibility for compatible classes.
• Human review only for breaking changes and privacy-impacting fields.
• Time-boxed responses (for example two business days) so boards cannot be silent vetoes.
• Federation (lesson 70) for cross-domain standards; domain owners for local fields.

A board that rewrites Learning's lesson titles does not govern schemas; it micromanages content.

Catalog requirements for an Active contract

Minimum catalog fields: contract id, version, owner team, owner contact, producer system, compatibility mode, output locations (topic/table/API), SLO summary, prohibited fields link, changelog, consumer list, status (Active/Deprecated), last CI green build.

If the catalog cannot answer "who do I page when enrollment_daily is wrong?", it is a search index, not governance.

Handling fan-out consumers you do not know

Lineage helps for jobs in the platform. It misses spreadsheets and undocumented scripts. Mitigations:

• Certified tag: only supported consumers get the tag and support SLO.
• Deprecation windows long enough for discoverable consumers; undocumented consumers get best-effort notice via catalog changelog.
• Monitoring for query patterns against columns marked deprecated.
• Explicit estate standard: certified paths must register as consumers in the contract file to receive break notices.

You cannot eternally support ghosts. You can refuse to call unregistered use "certified."

Money and units: a semantic contract checklist

Whenever a numeric field might be money, duration, or count, the contract must state: unit, currency if any, timezone if timestamp-adjacent, and rounding. Add a CI lint that flags field names containing amount, price, cents, seconds without a unit metadata key. This single lint prevents a class of "compatible but wrong" incidents that schema registries will never see.

TechLearningPro progress contract excerpt (richer)

yamlCopy
name: learning.progress
version: 12
status: active
owners:
  producer_team: learning-app
  contract_owner: learning-data
  consumers:
    - learning-data.lander
    - learning-data.completion
compatibility:
  mode: BACKWARD_TRANSITIVE
  registry_subject: learning.progress-value  # if Kafka path enabled
key:
  primary: event_id
  partition_hint: learner_key
event_time: event_time
idempotency_key: event_id
fields:
  - name: event_id
    type: string
    required: true
  - name: learner_key
    type: string
    required: true
    semantic: identity.learner_key
  - name: lesson_id
    type: string
    required: true
  - name: event_time
    type: timestamp
    required: true
    timezone: UTC
  - name: event_date
    type: date
    required: true
    derivation: "date(event_time) in UTC"
  - name: device_class
    type: string
    required: false
    default: unknown
    enum: [mobile, desktop, tablet, unknown]
prohibited_fields: [email, phone, payment_token, card_number, raw_ip]
landing_slo:
  p95_seconds: 900
notes:
  - Schema-valid rows may still fail quality gates (duplicates, unknown lessons).
  - definition_version lives on completion products, not on raw progress.

Consumers that need email for support use a different product with a legal basis—not this contract.

Dual-run parity query patterns

sqlCopy
-- Billing plan_code vs sku_code dual-run
SELECT
  COUNTIF(plan_code IS NULL) AS missing_old,
  COUNTIF(sku_code IS NULL) AS missing_new,
  COUNTIF(plan_code IS DISTINCT FROM sku_code) AS mismatched
FROM billing.enrollment_daily
WHERE event_date = DATE '2026-11-10';

Parity need not be zero mismatches if a documented mapping table exists; it must be explained. Unexplained mismatches block cutover.

Interview-ready story tying 43 → 69 → 70

Lesson 43: make one topic evolvable without breaking the morning job.
Lesson 69: make tables and APIs obey the same discipline with CI and ownership across Learning, Billing, and Identity.
Lesson 70: decide which domains own which products so contracts have accountable homes rather than a central team forging everyone's meaning.

If you only remember one chain in an interview, remember that one.

Best Practices

• One owner of record per contract; consumers listed explicitly.
• Schema plus grain plus semantics plus prohibited fields.
• Transitive compatibility for long-lived event subjects.
• Table contracts for every certified mart and shared dimension.
• CI fails on silent renames and on prohibited fields.
• Dual-run for breaks; never "flip Friday."
• Pin consumers to contract versions; avoid unbounded SELECT * in certified paths.
• Tie metric definition_version changes to contract releases.
• Announce with upgrade order and rollback.
• Keep lesson 43 local rules for topics; use this lesson for estate process.
• Lineage before drop.
• Treat semantic unit changes as breaks even when types match.

Common Mistakes and Anti-Patterns

Registry as junk drawer. Subjects without owners or compatibility mode NONE in prod.

Wiki-only contracts. No CI; the page is stale by construction.

Kafka-only governance. Warehouse and APIs remain wild west.

Rename in place. Breaks every unnamed consumer; use add/deprecate.

Compatible schema, broken meaning. Units and status codes changed quietly.

Consumer not listed. You cannot notify who you do not know; lineage is mandatory for certified datasets.

Gate disabled to ship. Contract tests skipped "just this once" become the permanent path.

SELECT * as integration. Any additive column becomes an accidental dependency.

definition_version as a comment. Not enforced in SQL or tests.

Platform team owns everyone's meaning. Central bottleneck and wrong semantics; platform owns tooling, domains own meaning.

Breaking change without dual-run. Instant cutovers on shared marts.

PII "temporary" fields in examples. Fixtures leach into prod contracts.

Troubleshooting and Design Review

CI green, prod consumers fail. Consumer used undocumented fields or SELECT *; shadow traffic or consumer packs missing. Add expectations for the real SQL.

Registry rejects a change you believe is compatible. Defaults missing; transitive history conflict; wrong compatibility mode. Pull full subject history.

Two contracts describe one table. Merge ownership; one document wins; the other becomes an alias with a sunset date.

Board wrong after "non-breaking" deploy. Check semantic version and cached extracts; invalidate on contract version.

Billing join fanout after Identity change. Key resolution contract drifted; verify grain of resolve API and whether one alias maps to many keys.

Design review:

• Who is the producer owner and who are the mandatory consumers?
• What compatibility mode is enforced in CI, and is it transitive for events?
• Where do table and API contracts live?
• What is the dual-run plan for the next breaking change?
• Which prohibited fields are scanned?
• How does definition_version appear in tests?
• What does lineage show before a drop?

Interview Questions and Answers

Q1. What is a data contract beyond a schema file?

It is the producer-consumer agreement: schema, grain, keys, semantics, compatibility policy, freshness, ownership, change process, and prohibited fields. A schema is necessary structure. Without the rest, compatible bytes can still be the wrong business fact.

Q2. How does this lesson relate to Schema Registry without duplicating Kafka training?

Lesson 43 covers subjects, ids, wire format, and compatibility on topics. This lesson reuses those rules for events and extends the same governance to warehouse tables and APIs, with CI, ownership, and breaking-change playbooks across TechLearningPro domains.

Q3. Give examples of compatible vs breaking changes.

Compatible: optional field with default; additive nullable warehouse column; new API field ignored by old clients. Breaking: rename; remove required field; incompatible type change; grain change; semantic unit change on the same field; required field without default when old writers still run.

Q4. Why are semantic breaks dangerous in interviews' eyes?

Because automated compatibility stays green. Cents versus dollars, or completion-v3 versus v4 rules, will not fail Avro resolution. I call out semantic versioning and board cutovers explicitly.

Q5. How do you test contracts in CI?

Compatibility against previous artifacts or registry; golden fixtures; prohibited-field scans; consumer expectation SQL/API tests; checks that metric version pins match contract releases.

Q6. Who should own the completion mart contract?

Learning-data as producer of the certified metric, with the ops board and learning-science as listed consumers, platform providing CI/catalog, privacy approving sensitive fields. Platform should not invent completion meaning alone.

Q7. How do you handle a required rename in Billing?

Add new column, dual-write, migrate consumers, parity check, then deprecate old column on a dated change. A single PR that renames in place fails review.

Q8. What is consumer-driven contract testing here?

Critical consumers contribute tests that run against producer staging or mocked contracts so producer changes fail before merge if they break those expectations—especially for Identity APIs and certified mart columns.

Q9. When is FULL_TRANSITIVE worth the pain?

When many consumers upgrade out of lockstep and history is long-lived, such as purchase or progress streams retained for restatement. Non-transitive checks allow type flip-flops across two hops that each looked local.

Q10. How do table contracts differ from event contracts operationally?

Events often enforce via registry serializers at write time. Tables enforce via migration CI, engine schema evolution rules, view pins, and publish gates. Both need ownership docs and breaking playbooks; only the enforcement knob differs.

Q11. What do you refuse to put in a mart contract?

Email, payment tokens, and other prohibited fields; unbounded SELECT * as the integration mode; dual certified definitions without version pins.

Q12. How would you explain contracts to a skeptical analytics lead?

Without them, every dashboard is a private fork of meaning. Contracts make changes reviewable, testable, and reversible—the same way APIs are treated in application engineering—so Monday's number has a named owner and a version.

Hands-On Exercises

Exercise: Versioned contracts for progress and completion

Problem statement. Create versioned data contracts for TechLearningPro progress events and published.completion_attempt. Define backward-compatible evolution for an additive field. Reject a breaking rename in CI. Include at least one warehouse/table contract test and one note on how lesson 43 registry rules apply if the optional Kafka path is used. Define ownership and a dual-run plan for a Billing plan_code → sku_code change.

Tasks.

1. Author YAML/JSON contract documents for the event and the mart table, including grain, prohibited fields, owners, and freshness.
2. Implement CI checks (script or pipeline stub) that allow additive optional fields and fail renames without a dual-run plan.
3. Add consumer expectation SQL for null keys and definition_version.
4. Write the Billing rename dual-run plan with parity query and consumer upgrade order.
5. Document what the registry will and will not validate if Kafka is in scope.
6. List three semantic breaks that would stay CI-green without extra tests, and add one such test.
7. Link contracts into a short catalog README used by the capstone repo.

Expected deliverables. Contract files, CI script or job definition, consumer tests, dual-run plan, catalog README, decision note on registry mode.

Evaluation rubric.

Deep dive: coordinating definition_version with schema versions

Structural schema version and metric definition_version move on different clocks. You might ship completion_attempt schema v4.2 with an additive column while remaining on definition_version=completion-v4. You might also ship completion-v5 with no structural change. CI must therefore track two channels:

1. Schema channel — field add/remove/type (compatibility engines).
2. Semantic channel — enum or metadata key definition_version / metric_semver.

Board cutovers always follow the semantic channel. Schema channel alone never authorizes a board meaning change. Capstone demos that show a new column but quietly alter completion rules without bumping semantic version fail this lesson's standard even if lesson 68's gate still passes structural checks.

Implementation sketch: store metric_semver: 4 in contract metadata; a test greps board SQL and runbook for completion-v4; a second test fails if transform code references v5 rules while metadata stays 4.

Deep dive: contract testing in multi-repo estates

Producers and consumers often live in different repositories. Options:

• Publish contract artifacts to an internal package registry on each Active tag; consumers depend on the package.
• Central contracts repo with CODEOWNERS per domain path; producers and consumers open PRs there first.
• Git submodules / sparse checkouts — workable, operationally heavy.

For TechLearningPro coursework, a single repo contracts/ folder is enough. In enterprises, the central contracts repo plus package publish scales better than copying YAML by hand.

Deep dive: Avro defaults versus JSON Schema and tables

Avro's reader/writer resolution makes defaults a first-class compatibility tool. JSON Schema validation in landers often treats missing fields as errors unless you configure defaults or use a transform step. Warehouse tables use NULL as the practical default. When one event becomes a table row, document the defaulting step: registry default "unknown" must become SQL COALESCE or load-time default consistently. Otherwise Kafka consumers see "unknown" while SQL analysts see NULL and dashboards diverge—another semantic break invisible to compatibility mode.

Putting it together for the capstone handoff

Before you leave lesson 69, update the lesson 68 repo (or your local capstone) with: Active contracts for progress and completion; CI job stub; Billing dual-run plan on file even if unused; ownership matrix; link from README to lesson 43 for optional Kafka. That handoff is the practical exam.

Lesson Summary

Data contracts turn schemas into estate-wide promises: who produces, who consumes, what may change, what must never appear, and how breaks are rehearsed. Lesson 43 remains the Kafka/registry mechanics reference; this lesson adds table and API contracts, CI, ownership, and semantic versioning beside definition_version. TechLearningPro's completion platform stays certified only when Learning, Billing, and Identity changes arrive through contracts rather than through quiet renames.

Next, Data Mesh and Domain-Oriented Data Products asks how far to push domain ownership—and when a centralized lakehouse from Module 5 remains the better product boundary.

Appendix: field-level ownership notes for shared dimensions

Shared dimensions such as lesson_id and course_id need a home. Learning owns the lesson dimension product; other domains may cache a projected read-only port but may not invent parallel ids. When Billing wants a SKU-to-course map, that map is a Billing product that references Learning's course_id contract, not a private course naming scheme. Document shared dimensions in the federation catalog with a single authoritative producer. Conflicts—"two course_id namespaces"—are resolved by federation, not by each dashboard picking a favorite. Capstone teams that fabricate a second lesson id to "make the join easy" fail contract governance even if row counts look pretty. Prefer an explicit mapping table with owners, versions, and validity windows. Invalidate caches when the dimension contract minor version increments if fields material to joins changed. Keep examples in the catalog so newcomers do not scrape CMS tables directly. Direct scrapes recreate the central-scrape problem lesson 70 tries to retire, only now with worse lineage. Finally, mark deprecated dimension attributes with sunset dates the same way you deprecate event fields, so BI extracts do not resurrect dead columns after a cleanup.

Appendix: sample breaking-change calendar

Week 0: proposal PR with lineage consumer list and privacy checklist.
Week 1: federation or designated approvers sign; Candidate contract published.
Week 2–3: dual-write in prod; parity dashboards live; consumers begin upgrade.
Week 4: consumer acknowledgment deadline; stragglers escalated.
Week 5: stop old write path; old read remains.
Week 8+: remove old field/table per retention; archive schemas.

Compress only when consumer count is one and tests are strong. Never compress by skipping parity. Record the calendar in the dual-run plan file so on-call knows whether today's traffic should still see both columns. Capstone grading can accept a simulated calendar with clear dates even if the wall clock is shorter, provided the artifacts exist.

Appendix: checklist before declaring a contract Active

• Owner team and contact populated
• Consumers listed or explicitly "none certified yet"
• Compatibility mode set and CI enforces it
• Golden fixtures present
• Prohibited fields listed and scanned
• Grain and keys documented
• Semantic/metric version rules documented if applicable
• Landing or serve SLO stated
• Catalog entry created
• Runbook link for contract failure
• Privacy checkbox completed
• Candidate soak evidence attached

Print this checklist in the PR template. Capstone reviewers may use it verbatim as an acceptance gate for the contracts portion of lesson 68/69 handoff.

Appendix: example CI log messages worth keeping

Good CI failures teach:

• BREAKING: removed field lesson_id; open dual_run_plan.md or restore field
• PROHIBITED: field email present in contract fields
• SEMVER: definition_version enum is completion-v4 but board SQL pins completion-v3
• OWNER: owners.producer_team missing
• COMPAT: mode NONE not allowed for status=active

Keep messages actionable. "Schema invalid" without a path wastes hours. Capstone reviewers should see at least one deliberate red CI log in your evidence pack for lesson 69.

Further Reading

• Schema Registry and Event Contracts for compatibility levels, subjects, and wire format.
• Confluent Schema Registry documentation for compatibility modes you may already run.
• OpenAPI Specification for API contract shape.
• Apache Iceberg Schema Evolution and Delta Lake column mapping docs for table-side evolution rules.
• Data Pipeline Testing and CI/CD for where contract tests sit in the pipeline.
• Data Lineage and Metadata Management for finding consumers before breaks.
• Capstone lesson 68 for the platform these contracts protect.