Introduction to Data Engineering

Data Engineering & Modern Data Platforms · Lesson 01 · Data Engineering Foundations

Lesson Overview

On the Monday after TechLearningPro's enrollment week, two numbers for the same sentence reached the same leadership meeting. Product analytics reported that 53 percent of newly enrolled learners completed lesson one. The lesson-completion warehouse reported 41 percent. Both queries ran. Both were "correct" against the tables they touched. Neither number could be defended, because nobody had written down what "completed" meant, which system was allowed to say it, or how late a progress event could arrive and still count.

That gap is the subject of this lesson, and of the course it opens. Data engineering is the discipline of building and operating the systems that turn operational facts into data other people can trust: analysts, instructors, finance, support, and models. Joe Reis and Matt Housley put the obligation cleanly in Fundamentals of Data Engineering: data engineering develops, implements, and maintains the systems and processes that take in raw data and produce high-quality, consistent information for downstream use. The definition is useful because it names an output (information someone can use) and a duty (the system keeps producing it after the first successful run). A notebook that answered the question once is analysis. A table that answers it again tomorrow, with a known grain, a known delay, and a known failure mode, is a data product. Data engineering is how that table comes to exist and stays honest.

This lesson is the home for Data Engineering & Modern Data Platforms. You will leave with a working vocabulary for the data lifecycle, for data products, and for the six-part decision lens this course uses whenever a design is proposed: correctness, freshness, scale, reliability, security, and cost. You will also see how this track differs from the primers already on TechLearningPro. /learn/python teaches the language. /learn/sql and /learn/postgresql teach queries and the relational engine. /learn/kafka teaches the log. This course assumes you can read those tools and spends its time on the decision they are serving: which fact is the source, which derived dataset is allowed to speak for the business, and what happens when a day has to be replayed.

The running system for the track is the TechLearningPro Learning Platform analytics estate: enrollment in the operational database, progress events from the lesson player, and a lesson-completion warehouse that instructors, product, and finance read. Later lessons open that estate at the seams that matter: architecture shapes, ownership, Python, SQL, ingestion, orchestration, Spark, Kafka, lakehouse formats, and production operations. Module case studies (an online retailer, a marketplace analytics slice) are staff-training designs with stated assumptions. They are not disguised product claims.

Planning assumptions used whenever this lesson quotes a number, labeled so they are not mistaken for a capacity study:

• About two million active learners in a normal month, with an enrollment-week spike.
• On the order of fifty million progress events on a peak day. Checkpoint events dominate; completion events are a small fraction.
• The player writes progress to PostgreSQL first. The warehouse is a derived store, not the system of record for "where the learner is in the lesson."
• Instructor studio and the Monday business review can tolerate data through the previous day. The player cannot. Those are different products.

Six modules, seventy lessons:

Lesson 2 draws the architecture shapes this lesson only names. Lesson 3 assigns owners. Lessons 4 and 5 build the Python you will use in pipelines. Lesson 6 builds the SQL that defines a metric without changing its grain by accident. This lesson only requires that you can point at a fact and say where it was born, who consumes it, and which of the six lens questions would make you reject a design.

Learning Objectives

By the end of this lesson you will be able to:

1. State what data engineering is for, in terms of a durable data product rather than a one-time query.
2. Separate data engineering from product software engineering, analytics, and data science on the same learning platform.
3. Walk a fact through the data lifecycle from generation to retirement, and name the undercurrents that apply at every stage.
4. Define a data product with a grain, a consumer, a freshness bound, and a source of record.
5. Apply the six-part decision lens (correctness, freshness, scale, reliability, security, cost) to a proposed pipeline.
6. Explain why two correct queries can publish two different completion rates, and what record would have prevented it.
7. Sketch the TechLearningPro analytics flow from the player and the enrollment database into a completion warehouse, and mark what that warehouse is not allowed to be.
8. Review a small Python contract for progress events and reject a batch that is not safe to curate.

Prerequisites

• You can read Python 3.11 or newer, including dataclasses and type hints. The contract checker in this lesson uses the standard library only. You do not need a data-science distribution.
• You have written SQL against a relational database, including a join and a filter. Lesson 6 will slow down and rebuild that skill for warehouse grain. This lesson only asks you to recognize a table.
• You have seen an application write a row or emit an event when a user does something. The examples assume HTTP, JSON, and PostgreSQL exist in the product. They do not assume a particular cloud or warehouse vendor.
• Familiarity with a command line is enough to run a single Python file. Orchestrators, Spark, and Kafka are later modules. Where this lesson mentions them, it says what decision they would serve, and points at the primer or the later lesson.

If you want the shorter primers first, start at /learn/python, /learn/sql, and /learn/postgresql, then return. This lesson will still be the one that asks which system is allowed to define completion.

Detailed Core Concepts

Purpose

Data engineering exists so that a decision made on Thursday uses the same meaning of the data as the decision made on Monday, after a backfill, a late event, and a new consumer. The product codebase already contains every event the player emits. It does not, by itself, tell an analyst which of those events is the business fact, which duplicates are retries, or which fields are personal data that must not land in a warehouse. The data engineer's job is to make those choices visible, attach them to a consumer who will suffer if they are wrong, and leave a path to replay the day when the choice changes.

Three purposes show up in every TechLearningPro data design review:

1. A stable meaning. "Completed lesson one" is a definition with inclusions, exclusions, and a grain. It is not whatever the last dashboard author typed.
2. A reliable delivery. The definition is recomputed on a schedule or a trigger, and a failed run is visible. A heroic notebook has no on-call story.
3. A boundary. Operational systems stay fast for the learner. Analytical systems stay explainable for the business. Copying every transactional table into a warehouse without a grain is not a boundary. It is a second, worse copy of the product.

A request that does not serve one of those three is often a query, and it should stay a query until a second consumer or a second week appears. Promoting every question into a pipeline is how platform teams drown. Refusing every promotion is how the company accumulates private spreadsheets that disagree.

What data engineering is not

The same enrollment week involves four kinds of work. Mixing them produces designs that optimize the wrong failure.

Product engineering and data engineering share a border at the event. If the player retries a checkpoint and emits two events with two identifiers, the warehouse cannot invent the idempotency key that should have been on the wire. If the warehouse silently drops one of them, the product team cannot see that a learner's progress was counted twice in a payout input. The contract is a joint artifact. Lesson 69 treats contracts as a governance practice. This lesson only insists that the identifier exists before anyone argues about tools.

Analytics and data engineering share a border at the metric. Analysts are often the first people who know the definition is wrong, because they sit with the decision. Data engineers are often the first people who know the definition cannot be computed from the events that exist. A healthy team writes the definition once, in a place both can read, and computes it in one job. A shadow SQL file in someone's home directory is a second product.

The lifecycle

Reis and Housley describe a lifecycle with stages and undercurrents. The stages are sequential only for a new fact. On a living platform they loop. This course uses five stages and treats the undercurrents as obligations on every stage, not as a phase at the end.

1. Generation. A source system creates a fact as part of doing its job. The enrollment service inserts a row when a learner pays. The player appends a progress event when a checkpoint is reached. Generation is owned by the product team. Data engineering does not get to wish a column into existence during a backfill. If the fact was never captured, the lifecycle starts with a product change, not a transformation.
2. Ingestion. The fact is copied, streamed, or captured into the analytical estate without becoming the system of record. Ingestion records when the copy happened, which source offset or watermark it reached, and what it did with a row it could not parse. A successful ingestion is not a successful metric.
3. Storage. The fact lands somewhere with a retention rule, a format, and an access boundary. Raw storage keeps the source's shape so a definition change can be replayed. Curated storage keeps the business shape so a consumer does not reimplement the definition. Mixing the two in one table is how raw payloads and "official" columns drift apart.
4. Transformation. A job applies the definition: filters, deduplication, joins to enrollment, and the rule that turns checkpoints into a completion. Transformation is where grain is either protected or destroyed. Lesson 9 and lesson 10 teach modeling. Here, grain means the one row identity that a consumer is allowed to count.
5. Serving. A person or a system reads the curated result: a BI tool, an instructor API, a finance extract, a feature table. Serving is a contract about freshness and schema. It is not an invitation to let the dashboard query the operational database "just this once."
6. Retirement. Events expire, experiments end, columns are removed, and a dataset loses its last consumer. Retirement is a stage. Dropping a table because a job failed and someone was in a hurry is not retirement. It is data loss.

The undercurrents run under all six:

• Security and privacy. Learner email is a generation fact in the identity store. It is not a warehouse column because a join was convenient.
• Data management. Names, ownership, retention, and quality rules live with the dataset.
• DataOps. The job is built, tested, deployed, and rolled back like software. Lesson 59 expands this. The seed is here: a pipeline without a version is a rumor.
• Architecture. The shape of storage and compute has to match the decision lens. Lesson 2 is that argument in full.
• Orchestration. Something notices that yesterday did not land. A cron line on a laptop does not.
• Software engineering. Types, tests, and review apply to pipeline code. Lessons 4 and 5 refuse the idea that data code is exempt.

A lifecycle diagram that draws boxes and forgets the undercurrents will still be presented as architecture. The Monday incident was not a missing box. The generation stage emitted events, ingestion copied them, and serving published a number. The missing piece was a definition at the transformation stage, owned by someone, with a grain.

Facts, events, and entities

Three words get used as if they were interchangeable. They are not.

An entity is a thing that has identity over time: a learner, a course, a lesson, an enrollment. You expect to update it. The operational database is usually the system of record for the current entity.

An event is a thing that happened, immutable once accepted: "learner lrn_1042 reached checkpoint 4 of lesson 7 at 14:02:11Z." You do not update an event to fix a definition. You append a correction, or you change the transformation that interprets the event. Progress events in this course are events. The player's current "resume position" is an entity derived from them, stored in PostgreSQL so the player can read it in milliseconds. Both can be true. They answer different questions.

A fact, in the analytical sense, is a business assertion at a grain: "this learner completed this lesson on this attempt." A fact is often derived. Treating a raw event as the fact is how retries inflate completion counts. Treating the fact as the source is how you lose the ability to change the definition.

The completion warehouse stores facts. The player stores a position entity. The event log, wherever it sits, stores events. When someone says "the data," ask which of the three they mean.

Data products

A data product is a dataset or a service with consumers, a contract, and an owner. The phrase is easy to cheapen. A table in a shared schema is not a product because it exists. It becomes a product when the following are written down:

• Name and purpose, in a sentence a consumer recognizes. "Lesson completions for enrolled learners, one row per learner, lesson, and attempt."
• Grain. The key. If two rows can exist for the same key, the product is not finished.
• Source of record for each input, and an explicit statement that this product is derived.
• Freshness. "Available by 06:00 UTC for the previous UTC day" is a contract. "Near real time" is not.
• Consumers. Instructor studio, the Monday metric review, finance if payouts ever depend on completion. A consumer list that is "everyone" has no one to ask when the definition changes.
• Quality rules that fail the job rather than warn forever. Duplicate event identifiers in one batch fail. A lesson id that does not exist in the catalog fails or is quarantined, and the quarantine is visible.
• Prohibited fields. Email, name, and payment tokens do not appear, even if the source payload had them.
• Owner. One accountable person or squad. Lesson 3 makes this operational. A product with a mailing list and no owner will be wrong in public.

The completion mart is the first data product in this course. Enrollment remains an operational entity. A nightly snapshot of enrollment, at a stated grain, can be a second product if finance needs "who was enrolled at midnight." The snapshot is not the same product as the live enrollment row. Naming them differently is the work.

Business value sits in the decision the product supports, not in the pipeline's complexity. For TechLearningPro the decisions are concrete. Content editors stop investing in a lesson that enrolled learners do not finish. Instructors see completion of their course without querying production. Finance, later, may use completion as an input to a revenue-share rule, which means the grain and the replay story become financial controls. A pipeline that does not name a decision is a cost center with a dashboard.

The decision lens

Every design in this course is evaluated on six questions. They are not a score you add up. They are the tensions you have to write down.

1. Correctness. Are the records complete, valid, deduplicated, and semantically the thing the consumer named? A completion built from any checkpoint event is complete in the sense that rows exist, and wrong in the sense that matters.
2. Freshness and latency. How late may the data be? The player needs the position entity now. The Monday review needs yesterday. Building a streaming completion mart for a daily meeting pays streaming complexity for a consumer who will not look until the morning. Freshness is a consumer property. It is not a virtue of the tool.
3. Scale. Volume, throughput, and growth, as planning assumptions with an owner. Fifty million events on a peak day is a reason to care about batch file size and partition strategy later. It is not, by itself, a reason to adopt a distributed engine on day one. Lesson 24 is where "it no longer fits one machine" becomes a design. Until the numbers are measured, they stay assumptions.
4. Reliability. Can the job recover, replay, and reject malformed records without corrupting the curated table? A reload of Tuesday that double-counts Monday's completions is a reliability failure even if the job is green.
5. Security and governance. Who can read the data, and which fields must never arrive? Access control that is "the warehouse role can see all schemas" is not governance. It is a postponed incident.
6. Cost and operability. Compute, storage, and the human cost of being woken up. A design the squad cannot replay on a weekday morning is more expensive than its cloud bill.

The lens is how you refuse a tool-first pitch. "We should put progress on Kafka" is not yet a design. "Instructors need completions within five minutes, the daily batch misses that bound, and we already operate a log" is the start of one. The second sentence can still be wrong. It is reviewable. /learn/kafka is where the log's mechanics live. Module 4 of this course is where the log enters the lifecycle. This lesson only gives you permission to ask the six questions first.

Source of record and derived data

A source of record (some teams say system of record) is the store whose value wins when copies disagree. For "is this learner enrolled right now," the enrollment database wins. For "where should the player resume," the progress table in that same operational estate wins. For "how many learners completed lesson one yesterday under the published definition," the completion warehouse wins, because the operational table does not store that definition. The warehouse does not win an argument about a single learner's live entitlement. If support reads the warehouse to decide whether a learner may continue, support is using a derived store outside its freshness contract, and the learner will be told something stale.

Derived data is allowed to be rebuilt. That permission is the point of keeping raw events. If the completion rule changes from "last checkpoint seen" to "last checkpoint seen and the practice score is at least the passing mark," the pipeline replays raw events. It does not update historical events in place. Consumers are told the definition version. A warehouse that cannot be rebuilt from raw inputs is a source of record that nobody chose, which is the worst kind: it cannot be corrected, and it was never authoritative.

History, only as far as it changes a decision

The analytical estate is older than the current tool advertisements. Operational reporting ran against the transactional database until those queries hurt the product, which is why a separate warehouse became normal. Bill Inmon described a top-down, integrated, historical warehouse. Ralph Kimball described dimensional models that serve a business process, with conformed dimensions so "learner" means one thing across facts. Both were responses to real failures of querying production, and both are still visible in how this course talks about grain and conformance. Lesson 10 teaches dimensional modeling. This lesson only borrows the warning: if two facts cannot share a definition of learner, the business will add the numbers anyway.

Lakes arrived because warehouses were awkward for raw, semi-structured, and large files, and because teams wanted to keep data before they knew every schema. Lakes failed in public when they became write-only swamps: files with no owner, no quality rule, and no serving contract. Lakehouse formats exist to put transactional guarantees and table semantics back on files in object storage. Streaming systems exist because some consumers cannot wait for a batch boundary. None of these sentences chooses an architecture for TechLearningPro. Lesson 2 does that choice explicitly. The historical lesson to keep is narrower. Each generation of platform was a response to a failure of the previous one, and each one reintroduced that failure when teams adopted the tool without the contract.

Architecture and Design Diagrams

The first diagram is a data flow for the learning platform's analytics estate. It is not a deployment diagram. It does not name a cloud, a warehouse vendor, or a topic. Those choices are lesson 2 and later modules. What it does name is the boundary between the system of record and the derived product.

mermaidCopy
flowchart LR
    subgraph product [Operational systems of record]
        player[Lesson player]
        enroll[Enrollment service]
        pg[(PostgreSQL: enrollment and live progress)]
        player --> pg
        enroll --> pg
    end

    subgraph analytics [Derived analytics estate]
        ingest[Ingestion: daily extract and event capture]
        raw[(Raw zone: progress events as received)]
        xform[Completion transform]
        mart[(Lesson completion warehouse)]
        ingest --> raw --> xform --> mart
    end

    pg --> ingest
    mart --> studio[Instructor studio]
    mart --> review[Monday business review]
    mart --> finance[Finance extract when contracted]

    player -.->|must not read| mart
    studio -.->|must not write| pg

Read the solid arrows as the copy that produces a data product. Read the dotted arrows as prohibitions. The player does not consult the warehouse to decide whether a learner may continue. Instructor studio does not "fix" a completion by writing to PostgreSQL. Finance reads the mart only after the grain is a financial input; until that contract exists, finance is not a consumer, and the diagram should not pretend they are.

A second picture is the lifecycle of one progress event that becomes, or does not become, a completion fact. This is the flow the Monday incident was missing.

mermaidCopy
flowchart TD
    emit[Player emits checkpoint or completion event]
    persist[PostgreSQL stores live position]
    copy[Ingestion copies the event into the raw zone]
    check{Contract checks pass?}
    quarantine[Quarantine with a reason]
    define[Transform applies the published completion definition]
    serve[Warehouse serves one row per learner, lesson, attempt]
    emit --> persist
    emit --> copy
    copy --> check
    check -->|no| quarantine
    check -->|yes| define --> serve
    quarantine --> replay[Replay after the contract or the source is fixed]
    replay --> copy

The live position and the warehouse fact diverge on purpose. A learner can have a resume position and zero completion rows. A replay repairs the warehouse. It must not move the learner backward in the player. That sentence is an architectural constraint discovered from the lifecycle, before any tool is chosen.

The decision lens sits on top of the same flow. A design review that starts with a product name has skipped the questions the arrows encode.

mermaidCopy
flowchart TD
    fact[Name the fact, the grain, and the consumer]
    lens[Score correctness, freshness, scale, reliability, security, cost]
    source[Name the source of record and the prohibited fields]
    shape[Choose a shape: batch, stream, or a query that is not a pipeline]
    record[Record the definition version and the replay path]
    fact --> lens --> source --> shape --> record
    record -->|definition changes or a lens answer was wrong| fact

Technical Explanation

Grain is the unit you are allowed to count

Grain is the combination of columns that identifies one row of a fact. For lesson completion in this course the grain is (learner_id, lesson_id, attempt_number). A count of rows is then a count of attempts that completed. A count of distinct learners per lesson is a different question and must be written as that question. If the transform emits two rows for the same triple, every downstream sum is a negotiation.

Grain failures are usually joins, not arithmetic. Enrollment is one row per learner and course. Progress events are many rows per learner and lesson. Joining them without aggregating events first multiplies completions by the number of matching enrollment columns someone did not expect, or, in the other direction, drops learners who completed a free preview that has no enrollment row. Both results have been presented as "the completion rate." Lesson 6 shows the SQL shapes. The technical point here is earlier: decide the grain before writing the join, and write down whether unenrolled preview activity is in or out of the product.

Attempt number has to come from somewhere. If the player does not emit it, the warehouse must not invent it from a timestamp sort and call the invention a source fact. Either the event contract gains attempt_number, or the product's grain stops including attempt and the definition says "first completion only." Inventing a business key in a transformation is how two replays produce two histories.

Time is part of the contract

Progress events carry at least two timestamps, and teams that keep only one will eventually answer the wrong question.

• Event time is when the learner reached the checkpoint, according to the player, in UTC.
• Ingestion time is when the analytical estate accepted the copy.

A daily completion mart for "yesterday" has to say which clock it uses. Event time matches the learner's experience and admits late data: an event that happened Tuesday and arrived Wednesday belongs to Tuesday, and Tuesday must be recomputed or the day is wrong. Ingestion time is easier to make idempotent and lies about Tuesday. The Monday meeting at TechLearningPro uses event time, with a restatement window of three days. That is a planning assumption, not a law of physics. The assumption is written on the product, because it forces the transform to be replayable for three days and forbids a job that only inserts "new" rows by ingestion time.

Timestamps without a zone are a defect. The contract checker later in this lesson rejects naive datetimes. A player that emits local time without an offset will be "correct" on one engineer's laptop and wrong for a learner in another zone. The standard is UTC on the wire. Display timezone belongs to the serving tool.

Identity, retries, and duplicates

The player retries when the network fails. A retry that creates a new event_id is a new event. A retry that repeats the same event_id is a duplicate delivery. The warehouse must treat those cases differently, and it can do so only if the producer and the pipeline agree.

• Same event_id, same payload: duplicate delivery. Keep one.
• Same event_id, different payload: a contract breach. Quarantine both versions and alert. Do not keep the "latest" by ingestion time and move on. Latest-by-arrival is how a corrupt retry overwrites a good event.
• Different event_id, same learner, lesson, checkpoint, and event time: a producer bug or a missing idempotency key. Quarantine is the safe default until product engineering confirms the key. Silently collapsing them hides the bug and will be wrong for a learner who genuinely repeated an action.

Deduplication is a correctness control. It is not a SELECT DISTINCT sprinkled at the end to make a dashboard stable. Distinct hides the grain problem. It does not record how many duplicates were removed, so the next incident has no count.

Correctness is several different promises

"Correct" is too large a word. Split it:

A job can be complete and still semantically wrong. That was the Monday incident: both pipelines ingested the same events and applied different semantics. Row-count monitoring would have stayed green. Definition versioning would have shown two versions in production, which is the actual fault.

Freshness is a distribution, not a slogan

State freshness as a bound a consumer can test. "The completion mart for calendar day D is published by 06:00 UTC on D+1, and days D-2 through D are restated if late events arrive inside the three-day window." The instructor studio can show "as of 06:00 UTC." It cannot show a spinner that implies the number is live.

Late events outside the window are a product decision. Extending the window forever makes every historical day mutable, which finance will reject once completions feed a payout. Closing the window at zero makes Tuesday wrong whenever the extract lags. Three days is the assumption this course will keep until a consumer signs a different one. The technical mechanism, a restatement job, is simple to describe and easy to get wrong: the restatement replaces the facts for those days at the agreed grain. It does not append a second copy. Replacement requires a transaction or a swap of a partition. Lesson 13 and the lakehouse lessons cover the mechanics. The requirement is set here.

Security is a field-level decision at ingestion

The operational progress row may sit beside a learner profile that includes email. The extract that feeds the raw zone is a projection, not SELECT *. The projection lists the columns the analytical estate is allowed to store: event id, learner id, lesson id, event type, checkpoint, attempt, event time, and a source watermark. Email, display name, and any payment identifier are prohibited. A prohibited field that appears in a payload is a failed batch, not a column you drop "later" after it has already been written to object storage. Copies are forever in a way that production rows are not, because backups and raw zones retain them after the product has deleted the account.

Learner id is still personal data. It is an internal identifier, not a secret, and it is enough to join a person back together if someone also has the identity table. Access to the warehouse is therefore granted to roles that need the mart, not to every employee, and the identity table is not placed next to the mart "for convenience." Lesson 61 is the privacy lesson. The ingestion rule is the part you can apply on day one.

Cost follows the lifecycle, not the invoice

A nightly batch that reads one day of events and rewrites three restatement days has a cost you can explain: storage of raw events for the retention period, and compute for the transform. A design that keeps every intermediate explode of a JSON payload in a proprietary warehouse, forever, has a cost the team will meet as a surprise. This lesson does not estimate a cloud bill. It requires the design to say which copies exist and how long they live. Raw progress events are retained for 400 days in the planning assumption, because definitions change and because a quarter-end question sometimes asks about last year. After that, the raw zone expires, and the curated facts remain under the finance retention rule if they have become a financial input. If they have not, curated completions follow the product retention schedule. Two retention regimes are normal. One unspoken regime is how data survives a deletion request.

Operability is the other cost. One squad owns the platform in the early planning assumption, the same shape as the product's early architecture. A lifecycle with a quarantine, a replay, and a single daily job fits that squad. A lifecycle with a streaming job, a backfill cluster, and three serving engines does not, until lesson 3's ownership map says who carries the pager. The lens question "can we operate this" is allowed to eliminate a technically elegant design.

What "done" means for a data product

A completion mart is done enough to serve the Monday review when all of the following are true:

• The definition is a versioned document, and the rows carry the version.
• The grain is enforced by a key the job can check.
• A replay of a closed day replaces that day's facts and changes nothing else.
• Quarantine counts are visible, and a non-zero quarantine fails the publish or is explicitly waived by the owner.
• The player and support do not read the mart for live decisions.
• Prohibited fields are absent, verified by the contract check, not by a promise.
• One owner can explain a row's lineage in a sentence: source event ids, definition version, event time.

A green job is not on that list. Green means the process exited. Done means a stranger can defend the number.

Practical Implementation or Design Walkthrough

Problem

The data guild will not let a completion number reach the Monday review until a batch of progress events passes a mechanical contract. The contract is the generation-to-ingestion boundary. It does not compute the completion rate. It decides whether a batch is safe to store in the raw zone and hand to a transform. The same checks apply to the staff-training retail case later in the lesson: different fields, same idea that a batch can be complete and still unacceptable.

Source data and target

Source: newline-delimited JSON written by the extract, one object per progress event, UTF-8. The extract is a projection of PostgreSQL. It is not a dump of the table.

Target: either an acceptance result with zero violations, or a rejection that lists every violation and writes nothing to the raw zone. Partial acceptance of a file is forbidden in this version. A file is a batch. Splitting good rows out and ignoring the rest hides the size of the defect. Quarantine of a whole batch is the behavior; a later lesson can introduce row-level quarantine once the volume justifies the extra path.

Planning assumption: a peak-day file is large, but this walkthrough processes an in-memory batch so the rules are obvious. Lesson 5 replaces the list with a generator. Do not "optimize" this checker by dropping a rule.

Contract

Semantic rule, still at the contract and not yet the mart: a completed event is allowed. It does not by itself mean the mart will emit a fact. The transform owns that. The contract only refuses events that cannot be interpreted later.

Implementation

Python 3.11 or newer. Standard library only. Save as progress_contract.py and run it with python progress_contract.py.

pythonCopy
"""Batch contract for TechLearningPro progress events.

A batch is accepted only when every event is valid and event_id is unique.
This module does not compute completion. It decides whether raw storage
is allowed to keep the batch.
"""

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Iterable, Mapping

ALLOWED_EVENT_TYPES = frozenset({"checkpoint", "completed"})
PROHIBITED_KEYS = frozenset({"email", "full_name", "card_token"})


@dataclass(frozen=True)
class ProgressEvent:
    event_id: str
    learner_id: str
    lesson_id: str
    event_type: str
    attempt_number: int
    event_time: datetime

    def __post_init__(self) -> None:
        if not self.event_id or not self.learner_id or not self.lesson_id:
            raise ValueError("event_id, learner_id, and lesson_id are required")
        if self.event_type not in ALLOWED_EVENT_TYPES:
            raise ValueError(f"unsupported event_type: {self.event_type}")
        if self.attempt_number < 1:
            raise ValueError("attempt_number must be >= 1")
        if self.event_time.tzinfo is None or self.event_time.utcoffset() is None:
            raise ValueError("event_time must be timezone-aware")
        if "@" in self.learner_id:
            raise ValueError("learner_id must not be an email address")


@dataclass(frozen=True)
class ContractViolation:
    event_id: str | None
    message: str


def _prohibited_key(payload: Mapping[str, object]) -> str | None:
    for key in payload:
        if key in PROHIBITED_KEYS or key.endswith("_email"):
            return key
    return None


def parse_event(payload: Mapping[str, object]) -> ProgressEvent:
    """Parse one object. Raises ValueError when the object cannot be a raw event."""
    banned = _prohibited_key(payload)
    if banned is not None:
        raise ValueError(f"prohibited field present: {banned}")
    raw_time = payload.get("event_time")
    if not isinstance(raw_time, str):
        raise ValueError("event_time must be an ISO-8601 string")
    parsed_time = datetime.fromisoformat(raw_time.replace("Z", "+00:00"))
    attempt = payload.get("attempt_number")
    if not isinstance(attempt, int) or isinstance(attempt, bool):
        raise ValueError("attempt_number must be an integer")
    return ProgressEvent(
        event_id=str(payload.get("event_id") or ""),
        learner_id=str(payload.get("learner_id") or ""),
        lesson_id=str(payload.get("lesson_id") or ""),
        event_type=str(payload.get("event_type") or ""),
        attempt_number=attempt,
        event_time=parsed_time.astimezone(timezone.utc),
    )


def review_batch(
    payloads: Iterable[Mapping[str, object]],
) -> tuple[tuple[ProgressEvent, ...], tuple[ContractViolation, ...]]:
    """Return accepted events and violations. Any violation means the batch is rejected."""
    accepted: list[ProgressEvent] = []
    violations: list[ContractViolation] = []
    seen: dict[str, ProgressEvent] = {}
    for index, payload in enumerate(payloads):
        try:
            event = parse_event(payload)
        except (ValueError, TypeError) as exc:
            event_id = payload.get("event_id")
            violations.append(
                ContractViolation(
                    event_id if isinstance(event_id, str) else None,
                    f"row {index}: {exc}",
                )
            )
            continue
        previous = seen.get(event.event_id)
        if previous is None:
            seen[event.event_id] = event
            accepted.append(event)
        elif previous != event:
            violations.append(
                ContractViolation(
                    event.event_id,
                    "duplicate event_id with a different payload",
                )
            )
        # Identical duplicate deliveries are kept once and are not violations.
    if violations:
        return tuple(), tuple(violations)
    return tuple(accepted), tuple()


def _demo() -> None:
    good = {
        "event_id": "evt_1001",
        "learner_id": "lrn_1042",
        "lesson_id": "les_07",
        "event_type": "completed",
        "attempt_number": 1,
        "event_time": "2026-03-02T14:02:11Z",
    }
    retry = dict(good)
    leaked = dict(good, event_id="evt_1002", email="learner@example.com")
    conflict = dict(good, event_type="checkpoint")
    accepted, violations = review_batch([good, retry, leaked, conflict])
    assert accepted == ()
    assert any("prohibited field" in item.message for item in violations)
    assert any("different payload" in item.message for item in violations)
    clean, none_found = review_batch([good, retry])
    assert none_found == ()
    assert len(clean) == 1
    assert clean[0].event_time.tzinfo is timezone.utc
    print(f"accepted={len(clean)} event_id={clean[0].event_id}")


if __name__ == "__main__":
    _demo()

Identical retries collapse to one event. A second payload under the same event_id fails the batch, including when the other row was a prohibited-field leak, because the function collects violations and then discards acceptance. That ordering is deliberate: a batch with any breach is not written. The demo asserts both the rejection and the clean path. There is no network call and no database. Lesson 5 will wrap the file read. Lesson 6 will turn accepted events into SQL.

What this walkthrough deliberately does not do

It does not choose Parquet, a warehouse, or a stream. It does not delete duplicates from a database. It does not define completion beyond refusing events that cannot support a definition. Those are later lessons, and pulling them forward would hide the contract under a tool decision.

How you would run it

Create a virtual environment only if you want isolation. The file imports nothing outside the standard library that ships with CPython 3.11+.

bashCopy
python3 progress_contract.py

Expected output is one line: accepted=1 event_id=evt_1001. A failure of an assertion means the contract rules were edited without updating the demo, which is the outcome the demo exists to catch.

Enterprise Real-World Scenarios

Two completion rates at TechLearningPro

Planning assumptions the analytics squad wrote after the incident, not before: the Monday review compares week-over-week completion of lesson one among learners who enrolled in the prior seven days; the number is a ratio of distinct learners, not of attempts; a learner counts as complete when a completed event exists for attempt 1, or when the highest checkpoint on attempt 1 equals the lesson's terminal checkpoint. The second clause exists because an older player build never emitted completed. Both clauses are part of definition version completion-v3. Version v2, still running in a personal dashboard, counted any event as completion. That is why 53 and 41 were both reproducible.

The repair was a lifecycle repair, not a dashboard repair.

• Generation: product engineering added attempt_number and a stable event_id on retries. Until that shipped, the warehouse quarantined events without an attempt number instead of defaulting it to 1.
• Ingestion: the extract became an explicit column list. A test payload with email fails in CI.
• Transformation: one job, definition completion-v3, grain (learner_id, lesson_id, attempt_number), restatement window of three days on event time.
• Serving: the Monday review reads only that job's output. The personal dashboard was pointed at it or turned off.
• Ownership: one squad is accountable for the mart. Analytics owns the wording of the definition and cannot ship a second job. Lesson 3 writes the RACI that makes this stick.
• Evidence that the repair worked: a replay of the incident week under v3 produces 41 percent, the 53 percent path is gone, and quarantine for the week is explained rather than empty by construction.

The player was unchanged for live resume. Support was told, in writing, not to read the mart to answer "can this learner continue." That prohibition is part of the data product, because the freshest completion fact can still be hours old.

Fieldnote Retail, an online retailer used for staff training

Fieldnote is a teaching boundary, not a TechLearningPro product. It sells goods online. The business question that broke was "net revenue for Tuesday." Finance reported one number from the order database. The marketing warehouse reported another. Both excluded tax. They did not exclude the same returns.

The lifecycle, with assumptions labeled:

• Generation. The order service writes an order entity in PostgreSQL when a shopper pays. The payments vendor, not Fieldnote, holds the card. A shipment event is born in the warehouse-management system. A return is born when a clerk accepts it, which may be ten days later.
• Volume assumption. About 80,000 orders on a normal day, 250,000 on a campaign day. Small for a distributed engine. Large enough to hurt the order database if finance runs a full-history scan at noon.
• Freshness. Finance closes the day at 08:00 UTC on the next morning and restates a day when a return arrives inside 14 days. Marketing's "same-day revenue" email is a different product and must be labeled as gross and provisional.
• Correctness. Grain of the finance fact is one row per order line per revenue event type (sale, return). A return does not delete the sale. It adds a negative fact. Deleting history makes Tuesday unreproducible.
• Security. Card tokens never enter the extract. Shopper email stays in the operational customer store. The mart carries shopper_id.
• Cost. A nightly batch plus a 14-day restatement fits the one analyst-engineer and one data engineer Fieldnote has in this case. A streaming revenue job would exist to feed an email that marketing has already agreed can be provisional and late. The lens says no.

The map the exercise asks you to draw is this lifecycle, with your own numbers. The lesson you should take from Fieldnote is the same one the completion mart taught: two correct queries over two grains are two products, and only one of them is allowed to be called net revenue.

Best Practices

• Name the source of record before you copy it. If you cannot say which store wins an argument, you are about to create a second one.
• Write the grain, the definition version, and the restatement window on the data product. A table name is not a definition.
• Project columns at ingestion. Prohibited fields fail the batch when they appear.
• Keep raw events immutable so a definition change is a replay, not an update of history.
• Separate the live product path from the analytical path in the diagram, including an explicit "must not read" arrow.
• State freshness as a clock, a deadline, and a late-data rule. Attach the rule to a consumer who agreed to it.
• Deduplicate on a producer-supplied identifier. Count the duplicates you collapse. Quarantine identifier collisions with different payloads.
• Prefer a query or a single batch job until a consumer's freshness bound forces more. Record the bound that would change your mind.
• Give every product one accountable owner. A shared Slack channel is not an owner.
• When two numbers disagree, compare definition version, grain, and time column before you compare tools.

Common Mistakes and Anti-Patterns

The warehouse is a copy of production. A nightly SELECT * into an analytical schema feels like progress and reproduces every operational quirk, every missing attempt number, and every email column. Analysts then build definitions in the BI tool, where they cannot be replayed together. The repair is a projection, a raw zone that keeps source shape on purpose, and a curated zone that is the only served definition.

Distinct as a correctness strategy. SELECT DISTINCT makes a chart stop doubling and destroys the evidence of why it doubled. The repair is a grain constraint and a metric of how many duplicate event_id values were collapsed. If the count is not zero, the producer is part of the incident.

One word for two clocks. Teams filter "yesterday" on whichever timestamp the first engineer selected. Late events vanish or land on the wrong day, and restatements fight inserts. The repair is event time for the business day, ingestion time for operational monitoring of the pipeline, and a written window in which event-time days are replaced.

Streaming because the tool is impressive. A completion rate reviewed once a week does not need a log consumer, a checkpoint store, and a new on-call skill. The repair is the freshness line of the lens, answered by the consumer. When the answer is five minutes, module 4 applies. When the answer is "before the Monday meeting," a batch job is the architecture.

Quarantine as a silent side path. Bad rows are written to a table nobody reads, the job stays green, and the completion rate quietly loses a segment of learners. The repair is a failing threshold: above a stated count, the publish does not happen, and the previous good day remains served with its timestamp visible.

The notebook is the pipeline. An analyst's file produces the number once, on a laptop, against a CSV someone exported. It cannot be replayed, reviewed, or secured. The repair is to treat a second consumer or a second recurrence as the trigger to promote the definition into the owned job, and to delete or redirect the notebook so it cannot diverge.

Retirement by accident. A raw prefix is cleaned up to save money, and with it the ability to restate definition v3 when finance asks why a quarter changed. The repair is a retention rule per zone, written next to the product, reviewed when the product becomes a financial input.

Troubleshooting and Design Review

Symptom: Two dashboards show different completion rates for the same week.

Likely cause: two definition versions, two grains, or two time columns. Less often, one dashboard is filtered to enrolled learners and the other includes previews.

Investigation: identify the dataset each dashboard reads. Compare grain, definition version, and whether the filter is "enrolled before the lesson event." Recompute both from the raw zone with one script.

Corrective action: serve one product. Retire the other. If both definitions are legitimately needed, name them as different products so nobody adds them together.

Symptom: Tuesday's completion count falls on Wednesday with no product change.

Likely cause: a restatement replaced Tuesday because late events arrived, and the new run applied a stricter contract that quarantined events the first run had accepted. Alternatively, the second run appended instead of replacing, and a downstream query that "dedupes" is now unstable.

Investigation: compare raw row counts for Tuesday's event time at the two publication times. Inspect quarantine counts. Check whether the curated key is unique after the second run.

Corrective action: make restatement a replacement at the grain. Publish the quarantine delta in the job log. If the contract tightened, replay the window on purpose and tell consumers the version changed.

Symptom: The player shows a lesson complete and the warehouse does not, or the reverse.

Likely cause: someone is comparing a live entity to a derived fact outside the freshness bound, or the player and the mart use different rules.

Investigation: read the learner's position in PostgreSQL and the event ids in the raw zone. Apply completion-v3 by hand. Note ingestion time versus event time.

Corrective action: if the mart is inside its contract, teach the reader which store answers which question. If the mart missed events the source has, the ingestion watermark is the defect. Do not "fix" the warehouse by writing to PostgreSQL.

Symptom: A batch file is rejected because of email, and the extract owner says the column is unused.

Likely cause: SELECT * or a view that gained a column when the profile table was joined "for debugging."

Investigation: read the extract SQL. Search the raw prefix for the key before you assume this is the first leak. If objects are already stored, treat retention and deletion as an incident, not a cleanup.

Corrective action: pin the column list, add the contract check to the extract job, and rotate any credential that was stored beside the dump. Then delete the leaked objects according to the privacy playbook. Lesson 61 holds the full playbook; the immediate move is to stop the copy.

Symptom: Peak day arrives and the daily job is still running at the Monday meeting.

Likely cause: the planning assumption of fifty million events was never measured, and the job reads raw JSON row by row on one small machine, or it joins to production.

Investigation: measure rows, bytes, and the slowest stage. Confirm the job does not query the operational primary.

Corrective action: meet the freshness bound with a larger single machine or a partitioned file layout before you adopt a new engine. Record the measured volume. If one machine cannot make the bound, lesson 24 is the next design, not a bigger guess.

A design review for this lesson asks six questions, in order. What is the fact, the grain, and the consumer? Which store is the source of record, and which fields are prohibited? What are the freshness bound and the restatement rule? What happens to a duplicate event_id and to a malformed row? Can the curated day be replaced from raw inputs? Who is accountable when the number moves? A review that starts with "why not Spark" has skipped the product.

Interview Questions and Answers

Q1. What is data engineering, in a definition you would defend to an analyst and to a product engineer?

Data engineering builds and operates the systems that turn raw operational data into information other people can use again tomorrow, with an explicit quality and freshness contract. Reis and Housley describe it as the discipline that takes in raw data and produces high-quality, consistent information for downstream use. In practice I name the data product: grain, source of record, freshness, prohibited fields, and an owner. I also name what the discipline does not do. Product engineering owns the live write the learner waits on. Analytics owns the decision and co-owns the metric wording. If I cannot point at a consumer and a replay path, I have a query, not a data product.

Q2. How is a data product different from a table in a warehouse?

A table is a storage artifact. A data product adds a contract: the grain you are allowed to count, the definition version, the consumers, the time bound, the failure behavior, and a single owner. A table created for one analysis and left in a shared schema will be reused with a different guess about duplicates and time. I would not call it a product until a second consumer can trust it without asking the author, and until a bad batch fails visibly instead of quietly thinning the metric.

Q3. Walk a lesson-completion metric through the data lifecycle.

The player generates progress events and stores the live position in PostgreSQL, which remains the source of record for resume. Ingestion copies a projected event into a raw zone, with a watermark. Storage keeps those events immutable for a stated retention. Transformation applies a versioned definition at the grain (learner_id, lesson_id, attempt_number) and replaces days inside the restatement window. Serving exposes the mart to instructor studio and the business review, not to the player. Retirement expires raw events on their schedule and keeps or drops curated facts based on whether they became a financial input. At every stage I can say who is allowed to read email: nobody in this estate.

Q4. Why can two correct queries produce two completion rates?

Because "correct" described execution, not semantics. One query counted any progress event as completion. The other required a completed event or a terminal checkpoint, on attempt 1, among learners enrolled in a window. Different grain, different definition version, possibly different clocks. The repair is one served definition. Monitoring row counts would not have caught this. Comparing definition versions would have.

Q5. How do you decide between a daily batch and a streaming pipeline?

I start from the consumer's freshness bound and the squad's ability to operate the design. A Monday review that needs yesterday by 06:00 UTC is a batch, plus a restatement window for late event time. A consumer who will take a wrong action if the data is an hour old might justify a stream, and I then ask whether the source even emits events reliably enough to support that bound. I do not choose streaming because the volume is "large" until I have measured that batch cannot meet the bound. Kafka skills live in /learn/kafka. The decision to put the log in the lifecycle waits for a freshness answer.

Q6. What is the difference between a source of record and a derived dataset?

The source of record wins when copies disagree, and it is written by the system that performs the business action. The enrollment row and the live resume position are sources of record. The completion mart is derived. It is allowed to be rebuilt from raw events, and it is not allowed to drive the player. A derived store becomes an accidental source of record when it cannot be rebuilt and people have started to trust it for decisions the contract did not cover. I treat that as a design incident.

Q7. How should duplicate progress events be handled?

On a producer-supplied event_id. An identical payload is one event, stored once, and the duplicate count is a pipeline metric. A different payload for the same identifier is a contract breach and the batch or the row is quarantined, not resolved by "latest write wins." A different identifier for what looks like the same action is a producer defect until product engineering confirms an idempotency rule. DISTINCT at the end of a report is not a handling strategy.

Q8. Where do privacy controls belong if the warehouse is easier to secure later?

At ingestion, as a projection and a reject rule. "Easier later" assumes the prohibited field was never copied. Raw zones and backups make that assumption false. I fail a batch that contains email, name, or a payment token, and I do not put the identity table in the analytical project so that joins are convenient. Access control on the mart still matters, because learner id is enough to reidentify a person when combined with other data. Later is for defense in depth, not for the first control.

Q9. A stakeholder asks you to query the production database for a quarterly completion report. What do you do?

I refuse the production scan as the reporting path and explain the risk to the player: a large analytical query contends with live progress writes. I offer the mart if the definition and freshness match the question, or a snapshot extract taken off a replica or an existing export, with a column list. If the mart's definition is wrong for the quarterly question, we change the definition and backfill. We do not teach finance to query PostgreSQL. The exception is a one-time incident investigation against a replica, with a time limit, and it does not become the pipeline.

Q10. What evidence, one month after launch, would convince you the completion mart is fit for purpose?

The Monday number is produced by one definition version, the restatement window replaces days instead of doubling them, quarantine is either empty or explained, prohibited-field checks have run on every batch, and the player path has no dependency on the mart. I would also want one deliberate replay of a day that matches the prior publication, and a written owner who can trace a single row to source event ids. A quiet pager is supporting evidence only if the consumer actually used the dataset. A green job with two dashboards still disagreeing is a failed launch.

Q11. How do data engineering and analytics divide ownership of a metric?

Analytics, with the business consumer, owns what the metric means and whether it still matches the decision. Data engineering owns the computation, the replay, the failure behavior, and the technical contract of the events. Neither side silently forks the SQL. A change to the meaning is a new definition version, a backfill, and a note to consumers. A change to the job that preserves the meaning is an engineering change. When those two changes ship in the same week without a version bump, the Monday incident returns.

Q12. When should you refuse to build a pipeline?

When there is a single consumer, a single use, and no recurrence, and a reviewed query against an existing product will answer it. Also when the source does not capture the fact, and the honest work is a product change rather than a guess in SQL. Also when the proposed pipeline's operability exceeds the squad, and a coarser freshness bound would fit. Refusal should offer the alternative: the query, the product change, or the simpler schedule. It should not offer a private notebook that becomes the pipeline anyway.

Hands-On Exercises

Exercise: Map the data lifecycle for an online retailer

Problem statement. Produce a lifecycle brief for Fieldnote Retail's "net revenue by day" question. Use your own volumes, freshness bounds, and window lengths. Do not copy the numbers from this lesson. The point is that every number has an owner and a clock.

Tasks.

1. Name the business decision and at least four consumers. Include finance and an operator. State who is explicitly not a consumer.
2. List the generating systems (orders, payments, shipments, returns) and the entity or event each one owns. Mark the source of record for "what did the shopper pay" and for "what left the building."
3. Define the grain of a finance fact and the grain of a provisional marketing fact. Explain why a return is a new fact rather than a deleted sale, or argue the opposite and record who is harmed.
4. Write the ingestion projection: included fields and prohibited fields. Include one rule that fails a batch.
5. Specify freshness, the clock (event time or processing time), and the restatement window for returns.
6. Draw a data-flow diagram with one "must not" arrow.
7. Apply the six-part lens in a short paragraph each. Label every volume figure as an assumption.
8. Describe one failure: a late return, a duplicate order event, or a leaked email. Say how the design detects it and what the consumer sees while it is broken.

Expected deliverables. A brief of at most three pages, the diagram, the grain and projection, the lens notes, and the failure note.

Evaluation rubric.

You pass when a reviewer can point to the clock used for "Tuesday" and to the field that must never land in the raw zone, without asking you to interpret a diagram legend.

Lesson Summary

Data engineering is how a company keeps a meaning of data stable enough to decide on, after the notebook is closed. The lifecycle—generation, ingestion, storage, transformation, serving, retirement—plus the undercurrents of security, ownership, and replay, is the map this course will keep returning to. A data product makes that map specific: grain, definition version, freshness, prohibited fields, and one owner.

The TechLearningPro completion mart and the Fieldnote revenue fact fail in the same way when those pieces are missing. Two queries run. Two numbers are published. Nobody can say which clock, which grain, or which version was served. The contract checker in this lesson is the smallest mechanism that refuses a batch the raw zone should not keep. It is not an architecture.

You now have the map of the seventy lessons and the difference between this track and the Python, SQL, PostgreSQL, and Kafka primers. Lesson 2 takes the flow you drew here and asks which platform shape—operational store, warehouse, lake, lakehouse, stream, or serving layer—is allowed to hold it, and what each shape costs when the Monday number has to be restated.

Further Reading

• Joe Reis and Matt Housley, Fundamentals of Data Engineering — the lifecycle and undercurrents this lesson uses, and a careful separation of the role from adjacent disciplines.
• Ralph Kimball and Margy Ross, The Data Warehouse Toolkit — grain, facts, and conformed dimensions. Lesson 10 applies the method. The toolkit's warning about grain belongs here.
• Martin Kleppmann, Designing Data-Intensive Applications — reliability, derived data, and the difference between a system of record and a copy. Chapters on replication and batch versus stream are the right depth after lesson 2.
• PostgreSQL 16 documentation — the operational store this course treats as the source of record for enrollment and live progress. See /learn/postgresql.
• Python 3 documentation — dataclasses, datetime, and the standard library used in the contract checker. The language primer is /learn/python.
• SQL primer — queries against the mart, taught properly in lesson 6 of this course.
• Continue this course: /learn/data-engineering/modern-data-architecture-overview, then /learn/data-engineering/data-engineering-roles-and-responsibilities.