Data Lineage and Metadata Management

Data Engineering & Modern Data Platforms · Lesson 22 · Data Ingestion, ETL and Orchestration

Lesson Overview

Finance asked which file produced a single order line whose cents did not match the provider. The Airflow grid from Advanced Airflow Operations showed that tlp_orders_daily succeeded. The warehouse showed the row. Nobody could answer, from a system of record, which landing object, which cleanse, and which column expression produced extended_amount_cents. A developer traced Python for an afternoon. The trace was true that afternoon and false the next week, after the function moved.

Lineage is the recorded answer to "where did this dataset come from, and what did we do to it?" Metadata is the description that makes the dataset usable without that afternoon: schema, owner, grain, freshness, classification, and the quality result. A catalog is where that description lives. A pipeline run is where lineage events should be emitted, because the run knows its inputs and outputs. Writing lineage in a slide deck produces a picture. Writing it from publish_freshness and from the SQL that builds the fact produces a graph you can query after the next deploy.

This lesson uses the public OpenLineage model: a job, a run, and datasets, with facets for schema and column lineage. Marquez is the reference backend that receives those events. DataHub is discussed as a catalog style that stores entities and aspects such as ownership, schema, and upstream lineage, using the public URN shape for datasets. Nothing here depends on a private API. The same names should appear when Module 3 runs the cleanse on Spark, so the graph stays one graph.

Airflow dataset URIs remain scheduling signals. They are not a column-level catalog. Both can exist. They answer different questions.

Learning Objectives

By the end of this lesson, you should be able to:

1. Separate technical lineage, business meaning, and operational metadata.
2. Model a pipeline step as an OpenLineage job and one execution as a run.
3. Name inputs and outputs as datasets with a namespace and a name.
4. Describe schema and column-lineage facets well enough to read an event.
5. Say what START, COMPLETE, FAIL, and ABORT communicate to a backend.
6. Configure the Airflow OpenLineage provider to post events to an HTTP endpoint.
7. Map an Airflow DAG and task onto a catalog data flow and data job without inventing a second grain.
8. Record ownership, classification, and grain in a catalog entry for fact_order_line.
9. Explain why lineage does not replace reconciliation, tests, or a data contract.
10. Keep dataset names stable so a Spark job and an Airflow task attach to the same nodes.
11. Identify broken lineage from SELECT *, undocumented enrichment, and renamed columns.
12. Hand Module 3 a naming scheme the Spark OpenLineage listener can share.

Prerequisites

• Apache Airflow Architecture and DAGs and Advanced Airflow Operations: task ids, dataset outlets, freshness.
• Data Cleansing and Transformation: the column-level transforms lineage must describe, including cents and SCD changes.
• Dimensional Modeling and Data Warehousing: grain and owners of facts versus dimensions.
• Data Quality Engineering: quality results are metadata, and they are not the same object as a lineage edge.

Detailed Core Concepts

Three kinds of "what is this table?"

Technical lineage is the edge from input dataset to job to output dataset, optionally down to columns. It answers impact: if the reseller file schema changes, which tables change? It answers trace: this fact row was produced by which job run?

Business metadata is meaning: grain sentence, owner, description, a link to the reconciliation definition, and whether country is type 2. A graph of tables without grain tells you files touched SQL. It does not tell you whether summing the table is legal.

Operational metadata is what happened lately: last successful run, published business date, row counts, quarantine counts, reconciliation difference, and a freshness timestamp. ops.dataset_freshness is already this. A catalog should show it beside the schema so an analyst does not have to open Airflow to learn that Tuesday is late.

One tool can store more than one of these. The mistake is to store only the graph and call the documentation problem solved.

OpenLineage objects

OpenLineage is a specification for events. Producers emit JSON. Consumers such as Marquez store them. The objects you need:

• Job. A stable name for the work, in a namespace. techlearningpro / tlp_orders_daily.load_fact_order_line is a good job name. It matches the DAG and task, so an operator can find it.
• Run. One execution of that job, with a runId. One Airflow task attempt maps naturally to a run. A retry is a new run of the same job, or the same run id if you deliberately correlate attempts. Pick one convention and keep it. Correlating to the Airflow run_id plus task id plus try number is explicit and debuggable.
• Dataset. A namespace plus a name. postgres://warehouse plus analytics.fact_order_line. The namespace distinguishes the production warehouse from a staging database that happens to use the same schema name.

Events have an eventType: START, RUNNING, COMPLETE, FAIL, ABORT. COMPLETE means the job finished and the outputs are the ones named. FAIL means it did not. Emitting COMPLETE from a task that swallowed a SQL error is false lineage, the same class of bug as a green Airflow task with an empty table.

A facet is a named extension on a job, run, or dataset. The ones that matter here:

• Schema facet on a dataset: column names and types at the time of the event.
• Column lineage facet on an output: for each output field, the input fields that fed it.
• Data quality facet on a dataset: assertions that ran and whether they passed.
• Ownership and documentation facets when you know them at emit time. Ownership often lives more naturally in the catalog as a curated aspect, because a pipeline run should not be the only writer of "finance owns the metric."

You do not have to emit every facet on day one. Emit job, run, inputs, outputs, and COMPLETE or FAIL first. Add column lineage for the money columns, because those are the ones people trace.

Naming is the whole design

Lineage graphs split when names split. Decide the scheme and write it down.

Use the same job name if the Spark submit replaces the body of cleanse_orders, so history stays on one node. Use a new job name if both implementations write in parallel during a migration, and add a catalog note that one is retiring.

Partition-specific landing names are appropriate for objects that are files. The warehouse table name stays unpartitioned in the dataset identity, with the business date in the run facets or in the operational metadata. If you create a dataset per day for the fact table, impact analysis cannot say "this job writes fact_order_line" without a union you will forget to maintain.

Airflow's scheduling Dataset URI from lesson 21, postgres://warehouse/analytics.fact_order_line, can match this naming on purpose. Scheduling identity and lineage identity do not have to be the same string, and life is easier when they are obviously related. The scheduling URI is one opaque string. The OpenLineage dataset is a namespace plus a name. Document the mapping in one module, tlp/lineage_names.py, and import it from the DAG and from any Spark config you generate.

What column lineage can and cannot say

An output field with a list of input fields says "these inputs were used." It does not say the formula. extended_amount_cents coming from quantity, unit_price_cents, and discount_cents is the useful edge. The formula quantity * unit_price_cents - discount_cents belongs in the catalog description and in the SQL CHECK constraint, which is executable. A lineage edge without the formula is still enough to answer "does a change to the discount column touch revenue?"

SELECT * and INSERT without a column list produce edges that are either missing or suspiciously wide. A renamed column looks like a drop and an add. Both hide impact. Lineage quality follows SQL discipline: name the columns.

Enrichment joins add edges to reference data. If the country alias table feeds country_code, the alias table is an input. Leaving it off the event means a bad alias update has no impact path. The SCD loader's output dim_learner has inputs of staging.learner_change and the previous dim_learner version. The previous dimension is an input. Jobs that read and write the same table should say so, or a reader will think the dimension is only a function of today's file.

Catalog entities, with DataHub as the concrete public shape

A catalog stores entities. In DataHub's public model, a dataset has a URN of the form:

textCopy
urn:li:dataset:(urn:li:dataPlatform:postgres,analytics.fact_order_line,PROD)

The triple is platform, name, and environment. Aspects hang off the entity: schema, ownership, documentation, glossary terms, and upstream lineage. A data job represents a task, a data flow represents a pipeline or DAG. The exact aspect names are part of DataHub's published metadata model. You do not need to memorize every aspect to use the idea: the dataset is an entity you can describe even when today's pipeline did not run, and lineage aspects can be filled from OpenLineage events by a connector.

Marquez stores the OpenLineage graph directly and is the reference implementation of the spec. A team can run Marquez as the lineage store and still keep business descriptions in a catalog, or use a catalog that ingests OpenLineage. TechLearningPro's requirement is the model, not a single vendor screen: stable names, owners, grain, and run-level edges.

Do not hand-draw the graph in the catalog UI as the primary path. The next deploy will drift. Emit events. Curate descriptions and owners as aspects people edit on purpose. The split is the same as code versus a README: generated truth for edges, reviewed truth for meaning.

Ownership

fact_order_line has a platform owner who owns the pipeline and a metric owner who owns the definition of revenue. OpenLineage ownership facets and DataHub ownership aspects can represent both if you use a type or a role, not a single "owner" field that two teams fight over. Write the roles in the catalog:

• Pipeline owner: data platform. Pages from Airflow. Changes the DAG.
• Metric owner: finance data partner. Approves the reconciliation definition and the freeze calendar.
• Steward for learner email: the privacy owner. Decides who can see the column.

Lineage does not enforce access. Classification metadata (email is personal data, extended_amount_cents is financial) is what an access process can read. Putting the email column on a wide-open feature mart and documenting it only inside a DAG docstring means the catalog cannot warn anyone.

Quality metadata is attached, not implied

A COMPLETE event does not mean the reconciliation passed, unless the task that emits COMPLETE is publish_freshness after the raise-on-diff SQL. Attach a data quality facet or a catalog assertion result: recon_diff_cents = 0, quarantine under threshold, row count. Analysts trust a table because the last run's checks passed, not because a node exists in a graph.

Lesson 23 turns those checks into tests you can run before deploy. The lineage event is the production record that the same checks ran for a business date.

Lineage is not a contract

A contract says what the producer promises: column, type, nullability, grain, and what happens when the promise breaks. Lineage says what happened on a run. You can have perfect edges into a table whose grain changed without a version bump. You can have a contract and no edges, which is how impact analysis dies. Keep both. The contract review changes the schema facet you expect. The event shows the schema you actually wrote. A diff between expected schema and emitted schema is a useful alert and a small piece of code, not a new platform.

Architecture and Design Diagrams

mermaidCopy
flowchart LR
    landing[s3 landing file] --> cleanse[job cleanse_orders]
    cleanse --> staged[staging.order_line_winners]
    feed[learner change feed] --> scd[job load_dim_learner]
    scd --> dim[analytics.dim_learner]
    staged --> factJob[job load_fact_order_line]
    dim --> factJob
    factJob --> fact[analytics.fact_order_line]
    factJob --> fresh[ops.dataset_freshness]

Each box is an OpenLineage dataset or job. The fact job's column lineage for extended_amount_cents points at the staged cents columns, not at the raw price string, once cleanse has already emitted its own column lineage from the raw string to cents. Impact still walks the path if both events exist.

mermaidCopy
flowchart TD
    dag[Airflow worker task] --> emitter[OpenLineage emitter]
    spark[Spark listener later] --> emitter
    emitter --> http[HTTP POST api/v1/lineage]
    http --> marquez[Marquez or compatible store]
    marquez --> catalog[Catalog aspects and search]
    people[Owners and grain text] --> catalog
    freshRow[Freshness and quality results] --> catalog

Edges are emitted. Grain and owners are curated. Freshness is operational metadata the catalog displays. None of the three is a substitute for the others.

Technical Explanation

An event you can read

This JSON follows the OpenLineage shape: event type, event time, run, job, inputs, outputs. Facet schema URLs evolve with the spec. Treat the field names namespace, name, runId, and eventType as the stable core, and confirm facet schema URLs against the spec version you pin.

jsonCopy
{
  "eventType": "COMPLETE",
  "eventTime": "2026-03-02T06:11:04.000Z",
  "run": {
    "runId": "9c1e6a20-1b2d-4e5f-8a11-0d4c2f6e7a10"
  },
  "job": {
    "namespace": "techlearningpro",
    "name": "tlp_orders_daily.load_fact_order_line"
  },
  "inputs": [
    {
      "namespace": "postgres://warehouse",
      "name": "staging.order_line_winners"
    },
    {
      "namespace": "postgres://warehouse",
      "name": "analytics.dim_learner"
    }
  ],
  "outputs": [
    {
      "namespace": "postgres://warehouse",
      "name": "analytics.fact_order_line",
      "facets": {
        "schema": {
          "fields": [
            {"name": "order_line_id", "type": "text"},
            {"name": "extended_amount_cents", "type": "bigint"}
          ]
        },
        "columnLineage": {
          "fields": {
            "extended_amount_cents": {
              "inputFields": [
                {
                  "namespace": "postgres://warehouse",
                  "name": "staging.order_line_winners",
                  "field": "extended_amount_cents"
                }
              ]
            }
          }
        }
      }
    }
  ],
  "producer": "https://github.com/OpenLineage/OpenLineage"
}

The column lineage says the fact cents come from the staged cents. The cleanse job emits a second event whose output staging.order_line_winners.extended_amount_cents has input fields quantity, unit_price text, and discount text, with the transformation described in the catalog as the decimal-to-cents function. Splitting the events by job matches the DAG. A single mega-event from file to fact hides which task to rerun.

producer identifies the library that emitted the event. Keep it accurate so you can tell Airflow-provider events from a hand-rolled emitter.

Airflow provider configuration

Install apache-airflow-providers-openlineage on scheduler and workers. Point transport at a collector. Marquez accepts OpenLineage on POST /api/v1/lineage.

iniCopy
[openlineage]
transport = {"type": "http", "url": "http://marquez:5000", "endpoint": "api/v1/lineage"}
namespace = techlearningpro

The equivalent environment variables are AIRFLOW__OPENLINEAGE__TRANSPORT and AIRFLOW__OPENLINEAGE__NAMESPACE. The provider's documented precedence prefers Airflow configuration. Do not depend on a default localhost URL inside a container image and then wonder why production events never arrive.

With the provider enabled, task runs emit job names derived from the DAG and task. Verify one COMPLETE in the collector for load_fact_order_line after a staging run. If the HTTP call fails, decide whether lineage failure fails the task. TechLearningPro logs and metrics a failed emit and does not fail the warehouse load, because a Marquez outage should not block finance. The freshness row remains mandatory. Lineage is best-effort only if the gap is visible on a dashboard. A silent best-effort is how you discover a month later that the graph stopped in March.

The provider can attach operator-specific facets. A SQL operator may include the SQL text. That text can contain data if someone concatenates literals. Prefer bound parameters. Treat lineage storage as a place where SQL and sometimes sample values accumulate, and restrict access the way you restrict warehouse access.

There is a provider setting, spark_inject_parent_job_info, to pass parent run information into Spark submits so the Spark application links as a child of the Airflow task. Use it when cleanse moves to Spark. Confirm property names in the OpenLineage Spark documentation for the version you install. The listener class in wide use is io.openlineage.spark.agent.OpenLineageSparkListener, registered with Spark's spark.extraListeners, with spark.openlineage.transport.type=http and a namespace of techlearningpro. The point of those settings is shared names, not a new lineage product inside the cluster.

What to curate by hand

In the catalog entry for analytics.fact_order_line:

• Grain: one row per sold order line.
• Owners: pipeline and metric, as roles.
• Column descriptions: extended_amount_cents is quantity times unit price minus discount, in cents, half-even from decimal text.
• SCD note on learner_key: points at the learner version, not at "current country."
• Classification: no email column on this fact. Email lives on the learner dimension and is personal data.
• Link: reconciliation SQL path in the repo and the freshness dataset.

In the entry for staging.learner_change:

• Grain: one observed change.
• Note: snapshot sources are daily diffs and miss intra-day reversions. That sentence is metadata. It prevents a bad impact conclusion later.

Review catalog text in the same pull request that changes grain or a money formula. A lineage event will not update the English grain sentence for you.

Freshness as metadata, not as a guess from COMPLETE

Push or pull ops.dataset_freshness into the catalog on a schedule: business date, published_at, recon_diff_cents, quarantine cents. A catalog page that shows schema and a green "last job succeeded" from a different DAG is how a late fact table looks healthy. Bind the operational widget to the freshness row for that dataset name, the same name you used in OpenLineage.

Schema changes

Adding a column is an event with a new schema facet. Removing a column is an impact question: query the graph for downstream datasets that reference the field. If you never emitted column lineage, you only know which jobs read the table, and a person reads SQL. Emit column lineage for any column a downstream mart selects by name. Accept that a consumer using SELECT * outside your platform will not appear. Lineage sees producers and consumers you instrument. An external spreadsheet will not show up until someone records it. That is an argument for fewer uninstrumented copies, not for pretending the graph is the universe.

Practical Implementation or Design Walkthrough

Problem. Make tlp_orders_daily explain a fact row without an afternoon of reading Python, and make the catalog state the grain.

Steps.

1. Add tlp/lineage_names.py with constants for the warehouse namespace, the fact dataset name, the staging dataset name, and the job names. Use those strings if you build any custom facet.
2. Install the OpenLineage provider. Set transport and namespace = techlearningpro in staging first.
3. Run one DagRun. Find a COMPLETE event for load_fact_order_line and a FAIL event when reconcile raises. Confirm the failed run does not claim the fact output as a successful publish. If the provider emits outputs on the failing SQL task, that is acceptable as "this task attempted these datasets." The publish task's COMPLETE is the one the catalog should treat as the published version. Configure the catalog view to key off publish, not off every task.
4. Add a column-lineage facet for extended_amount_cents if the provider does not extract it from your Python merge. A small explicit emit from the merge function is better than a wrong automatic guess. Keep the emit behind the same function that writes the row, so you do not emit COMPLETE when the transaction rolls back.
5. Write the catalog descriptions for the fact, the learner dimension, and staging.learner_change, including the snapshot limitation.
6. Point the catalog's freshness panel at ops.dataset_freshness.
7. Change the discount rule in a branch and show the column-lineage edge still lists discount as an input. If it does not, the facet is stale and the emit path is wrong.

Acceptance. From a fact column, a person reaches the staging column and the job run id, and the run id matches an Airflow task attempt. The catalog grain sentence matches lesson 10. A failed reconcile does not move published_at.

Enterprise Real-World Scenarios

Tracing a one-cent drift to a float that lineage made visible

A provider mismatch of a few cents survived reconciliation because it sat under a threshold someone had raised during an incident and forgotten. The column-lineage facet for unit_price_cents still pointed at a Python job, and the schema facet showed the staging column as double precision for one environment that had not picked up the bigint migration. The catalog diff between expected schema (bigint cents) and emitted schema (double) was the alert. The code fix was the lesson 18 decimal parse. The metadata fix was to treat the expected schema as an aspect you review, and the emitted schema as an observation. Without the facet, the investigation started from "Airflow is green" and stayed there. Lineage did not compute the cent. It shortened the search to the job that still declared a float.

Two owners and a personal-data column

Product built a feature table by joining fact_order_line to dim_learner and selecting email "just for debugging," then published the table to a broader role. The fact's classification said it contained no email. The dimension's classification said email was personal. The feature job's lineage, once emitted, showed email flowing into features.order_debug. The privacy steward had a query for downstream datasets of that column and found the table the same day the job shipped. The repair was to drop the column and to keep the debug extract inside a role that already sees the dimension. The catalog did not block the SQL. It made the copy visible. A second control, outside this lesson, is permission on the dimension itself. Lineage is how you notice a permitted principal creating a less-protected copy.

One graph when Spark takes the cleanse

The cleanse task moved to Spark in Module 3. The first version emitted lineage under the namespace spark and the dataset name winners, while Airflow still emitted staging.order_line_winners in postgres://warehouse. The UI showed two islands. Renaming the Spark outputs to the shared scheme, and setting the parent job to tlp_orders_daily.cleanse_orders in namespace techlearningpro, joined the islands. The Airflow task remained the parent run. The Spark application was the child that actually touched the files. Impact analysis for the reseller landing path showed both. The naming module from the walkthrough was the artifact that prevented the split. A wiki diagram of "the Spark box" would have been outdated the day the job id changed.

Best Practices

• Name datasets as namespace plus stable name. Put dates in partitions and run metadata, not in the warehouse table's identity.
• Name jobs after the DAG and task so Airflow and the lineage store match.
• Emit COMPLETE only after the commit that made outputs visible. Emit FAIL when the task fails.
• Include column lineage for money and keys even if you skip it for low-risk columns at first.
• List reference data and the current dimension as inputs when the job reads them.
• Keep a single Python module of names shared by Airflow and future Spark jobs.
• Curate grain, role-based owners, and classification in the catalog. Do not expect a run event to be the editor of record for English definitions.
• Surface ops.dataset_freshness in the catalog next to the schema.
• Restrict access to stored SQL and any sampled values in the lineage backend.
• Fail visibly when emits stop. Do not fail the finance load on a collector outage unless you have chosen that tradeoff explicitly.
• Review catalog text in the pull request that changes grain or a formula.
• Prefer instrumented jobs over hand-drawn edges.
• When a consumer is outside the platform, record it as a manual dataset with an owner, or accept that impact analysis stops at the boundary and say so.

Common Mistakes and Anti-Patterns

A slide as the lineage system. It is outdated on the next pull request. Emit events from the run.

A new dataset name per environment stuffed into the table name. fact_order_line_prod and fact_order_line fork the graph. Use the namespace or the catalog environment field for prod versus dev.

SELECT * in the published SQL. Column lineage becomes "everything" or "nothing." Name columns.

Emitting COMPLETE before commit. Readers follow the event to data that rolled back. Emit after the transaction succeeds, in the same task that commits.

One owner field for two accountabilities. Pipeline failures page the wrong team, or metric changes ship without finance. Use roles.

Storing emails in lineage samples to "help debugging." The lineage database becomes another personal-data store with weaker controls. Schema and hashes are enough.

Treating an Airflow scheduling Dataset as column lineage. It is a URI that fires a DAG. Keep it. Also emit OpenLineage if you need columns and impact.

Ignoring the previous dimension as an input to an SCD job. The graph says the dimension comes only from today's file, so a bug in yesterday's version looks impossible. Declare the read.

Spark jobs in a private namespace. The parent Airflow task and the Spark job never meet. Share techlearningpro and the dataset names.

Catalog descriptions that contradict the CHECK constraint. Analysts follow the prose. Generate or review the prose when the constraint changes.

Green lineage and a stale freshness panel sourced from a different DAG. Operational metadata must name the same dataset as the edge.

Troubleshooting and Design Review

Design review:

1. Can you point to the module that defines dataset namespace and name?
2. Which task emits the publish COMPLETE, and is it after commit?
3. Do money columns have input fields listed?
4. Are SCD jobs honest about reading the current dimension?
5. Are pipeline owner and metric owner both present, with classifications on personal columns?
6. Does the catalog freshness panel read ops.dataset_freshness?
7. Will the future Spark listener use the same names?

Interview Questions and Answers

Q1. What is the difference between lineage and a catalog description?

Lineage is evidence from runs: this job version read these datasets and wrote those, perhaps at column level. A catalog description is curated meaning: grain, owners, and what a column denotes. You need the event to do impact analysis after a change, and you need the description to know whether the table is legal to sum. A graph without grain is an inventory of SQL. A README without events is stale.

Q2. What do OpenLineage job, run, and dataset each represent?

A job is the stable work, such as tlp_orders_daily.load_fact_order_line in namespace techlearningpro. A run is one execution, with a runId you can tie to an Airflow attempt. A dataset is a namespace plus a name, such as postgres://warehouse and analytics.fact_order_line. Events connect a run of a job to input datasets and output datasets. The names are the design. The transport is just HTTP.

Q3. What should a COMPLETE event guarantee?

That the job finished and the outputs it lists were made visible by a successful commit. It should not be emitted from a finally block that also runs on failure, and it should not be emitted before the transaction commits. FAIL covers the other path. A collector outage policy is separate: you may choose not to fail the task, and you still must not emit a false COMPLETE.

Q4. How do you configure Airflow to send OpenLineage events?

Install apache-airflow-providers-openlineage on scheduler and workers. Set [openlineage] transport to the HTTP JSON the provider documents, including the collector URL and endpoint such as api/v1/lineage for Marquez, and set [openlineage] namespace. Environment variables AIRFLOW__OPENLINEAGE__TRANSPORT and AIRFLOW__OPENLINEAGE__NAMESPACE match that configuration. Confirm a real event in the collector on a staging run before you trust the graph.

Q5. Why is a dataset-per-day name a problem for the fact table?

Impact and ownership want one entity, analytics.fact_order_line. A dataset named with the business date fragments the graph into a stream of near-duplicates and breaks a consumer that references the table. Put the date in the landing partition name and in the freshness row. Keep the table identity stable. The Airflow scheduling Dataset URI should stay stable for the same reason.

Q6. What does a column-lineage facet add beyond table-level edges?

It says which output fields depend on which input fields, so a change to discount is visible as an input to extended_amount_cents without reading the SQL under pressure. It does not replace the formula or the CHECK constraint. SELECT * and unnamed inserts make the facet useless. Name columns in the published SQL, and list reference inputs such as the country alias table.

Q7. How should an SCD load represent inputs?

Include the change staging table and the dimension table it reads to find the current version. A graph that shows only the day's file hides the dependency on yesterday's surrogate and effective dates. The output is the same dimension, with a new version row. The catalog text still carries the type-2 policy and, for snapshot sources, the intra-day history you cannot see.

Q8. How do DataHub-style URNs relate to OpenLineage names?

OpenLineage identifies a dataset by namespace and name in an event. DataHub identifies a dataset entity with a URN that includes platform, object name, and environment, such as urn:li:dataset:(urn:li:dataPlatform:postgres,analytics.fact_order_line,PROD). A connector maps one onto the other. Aspects such as ownership and documentation live on the entity even when no run happened today. Keep the object name identical to the OpenLineage dataset name so the map stays boring.

Q9. Why are there two owners on fact_order_line?

The platform owner runs the pipeline and answers pages about failed tasks. The metric owner defines revenue, the freeze calendar, and whether a reconciliation exception is acceptable. One "owner" field collapses those responsibilities and pages finance for a worker outage or lets a formula change ship without the metric owner. Role-typed ownership matches how the teams already work.

Q10. A job COMPLETED and the flash report is wrong. What do you look at besides the graph?

The freshness row and the reconciliation result for that business date, the schema facet versus the expected cents type, and the column definition in the catalog. Lineage can show that the job wrote the table and that cents came from staging. It will not show that someone raised the quarantine threshold. Quality metadata has to be attached to the run. Green edges are not a reconciliation.

Q11. How do you keep Spark and Airflow in one graph?

Use the same namespace techlearningpro, the same dataset names the warehouse already uses, and the same job name as the Airflow task whose body Spark replaced. Pass the parent job namespace, name, and run id into the Spark OpenLineage integration so the application is a child of the task. Register the Spark listener in the Spark configuration for that submit. A second naming scheme in the Spark repo will draw a second island the day it ships.

Q12. What should you refuse to put in the lineage store?

Raw personal data "as a sample," passwords, and full production extracts. SQL text can be useful and can also contain literals you did not mean to publish. Store schema, dataset names, job names, run ids, and column edges. Restrict the lineage database as you restrict the warehouse. Lineage is a map, not a second data lake.

Hands-On Exercises

Exercise 1. Name the graph

Problem. Write lineage_names and a sample COMPLETE event for the fact load.

Tasks.

1. Define namespace and name for the landing file pattern, staging winners, the learner dimension, and the fact.
2. Define the job name for cleanse, SCD, and fact load.
3. Draft the event JSON with inputs, outputs, and column lineage for extended_amount_cents.
4. State when in the task you emit it.
5. Explain how a snapshot-based learner load lists the current dimension as an input.

Acceptance. A reader can join the job name to a task id and the dataset name to a table. The event does not claim success before commit.

Exercise 2. Catalog the fact and spot a bad copy

Problem. Write the catalog entry and a second event that leaks email.

Tasks.

1. Record grain, both owners, and classifications for fact_order_line and dim_learner.email.
2. Add a fictional feature job event that outputs email from the dimension.
3. Describe the impact query a steward runs.
4. Point freshness at a concrete row shape: business date, published_at, recon diff.
5. Note the naming change required so a Spark cleanse does not fork the graph.

Rubric.

Lesson Summary

Lineage records which job run read which datasets and wrote which columns. The catalog records what those datasets mean, who owns them, and how sensitive they are. Operational freshness says whether the business date is actually published. OpenLineage is the event shape: job, run, dataset, event type, and facets. The Airflow provider can post those events to a collector such as Marquez. A DataHub-style catalog can hold the curated aspects and the URN for the same table name.

Names are the design. techlearningpro and analytics.fact_order_line should mean the same thing when Spark starts emitting events in Module 3. Column lineage for cents, honest inputs for SCD, and a freshness panel bound to ops.dataset_freshness are the parts analysts and incident responders actually use. The next lesson tests the transforms and the DAG so the graph describes code that has already passed a gate, not only code that happened to run.

Further Reading

• OpenLineage, specification — event types, jobs, runs, datasets, and facets including column lineage.
• OpenLineage, Airflow integration and the provider configuration reference — transport and namespace.
• Marquez, the reference OpenLineage backend, including POST /api/v1/lineage.
• DataHub, dataset URN and entity/aspect model in the public metadata documentation — for catalog identity as distinct from a single run event.
• OpenLineage Spark integration — OpenLineageSparkListener and parent job properties, for the Module 3 bridge.
• TechLearningPro, Advanced Airflow Operations — the publish task that should be the success signal you align with COMPLETE.
• Next: Data Pipeline Testing and CI/CD, then Big Data Fundamentals.