Data Lakehouse Architecture

Data Engineering & Modern Data Platforms · Lesson 49 · Modern Data Platforms and Cloud

Lesson Overview

The marker from Data Lake Architecture closed hour 13. Compaction then rewrote that hour into a new run prefix and published a second marker. A Spark restatement pinned to the first marker and a notebook that listed the parent hour counted different completions for Tuesday. Both numbers parsed. Neither reader had a snapshot. The orchestrator was supposed to be the only writer, and a backfill notebook was a second writer anyway. The JSON file had no compare-and-swap, so the later put won and the Spark job never heard about it.

Modern Data Architecture Overview defined a lakehouse as table semantics on files: a catalog of tables, a commit readers can refresh onto, schema enforcement, and a restatement that replaces Tuesday as a version rather than as a folder of hope. This lesson is that definition made operational for TechLearningPro. It decides which datasets stay prefixes, which become tables, and which questions still belong in a warehouse engine. It does not implement Delta's log, Iceberg's manifest tree, or Hudi's timeline. Those are lessons 50 and 51. It does not stand up a Databricks workspace or a Snowflake account. Those are lessons 52 and 53.

The failure to avoid is the slogan. "We are a lakehouse" after a bucket rename does not create a commit. A warehouse full of SELECT * copies does not become reliable because a slide drew a lake beside it. The work is a serving table with one grain, one writer protocol, and a version a reader can name.

Learning Objectives

By the end of this lesson you will be able to:

1. Define a lakehouse as files plus a metadata commit that readers pin, and separate that from a catalog over a live prefix.
2. Compare lake, warehouse, and lakehouse on correctness, freshness, concurrency, and cost for one TechLearningPro question.
3. Map lesson 13's zones onto table layers without inventing a second architecture under a new set of metal names.
4. State what optimistic commit on object storage guarantees, and what a PostgreSQL transaction still does that the log does not.
5. Choose a single serving system for completion-v3, and treat a second copy as a replica with its own clock.
6. Keep the lesson 47 prefix clock distinct from the table snapshot clock.
7. Decide when the PostgreSQL mart remains enough, using the lens from lesson 2.
8. Write the format-selection criteria that lessons 50 and 51 will apply, without pretending the formats are interchangeable in their operations.

Prerequisites

• Lesson 2: the shapes and the rule that a shape must change a decision-lens answer. The publication row is the ancestor of the snapshot id.
• Lesson 8: what a database transaction isolates. The lakehouse commit is easier to over-claim if you have not seen a real lock and a real snapshot.
• Lesson 10: facts, dimensions, and grain. The lakehouse table serves that model. It does not dissolve it.
• Lesson 13: zones, ELT, definition_version, and replace-the-window. The table commit is how the replace becomes atomic for readers.
• Lesson 17: the curated gate still runs before a version is the one Monday reads.
• Lesson 25: why object-store commit is not an HDFS rename. The table log is the designed answer to that gap.
• Lesson 47: prefix freshness versus the commit you must not quote as the same number.
• Lesson 48: run prefixes, markers, single-writer assumption, and the races that end this lesson's patience with JSON.

Detailed Core Concepts

The commit is the product

A lakehouse stores table data as files in the lake's object storage and stores the table's current version as metadata those files do not carry by themselves. A reader that opens version N sees the files version N names. A writer creates version N+1 by writing new files and committing metadata that points at the new set. Readers still on N do not observe a mixture of N and N+1. Compaction that rewrites files creates another version whose data values match the previous version. That is the property the hour-13 incident did not have.

The metadata is the table format. Delta Lake records a transaction log. Apache Iceberg records snapshots, manifest lists, and manifests. Apache Hudi records a timeline of instants. Lessons 50 and 51 open those structures. At this altitude they are the same promise: a version, a file list for that version, a schema, and a conflict rule when two writers commit together. A Hive table whose LOCATION is s3://tlp-raw/learning/progress/ has a schema and no version. Lesson 48 already said so. Promoting that table into a slide titled lakehouse does not add the log.

The catalog still matters. Something must answer "where is the current metadata pointer for learning.completion?" A metastore, Glue, Unity Catalog, or an Iceberg REST catalog holds that pointer. The pointer moving is the commit other engines can see. A pointer that is only a version-hint file overwritten on S3 is a weak commit. Lesson 51 will refuse that catalog for production. This lesson's rule is: if two engines can disagree on which metadata file is current, you do not yet have one table.

Lake, warehouse, lakehouse on one question

The question is Monday's completion count for a course, under completion-v3, restatable for three days, with a studio API that must not read a half-replaced day.

The warehouse row is the Kimball and Inmon discipline from lesson 10, running in an engine that enforces schema on write and runs SQL the analyst already knows. Snowflake, in lesson 53, is one such engine. PostgreSQL can be a small one, which is the choice lesson 2 left in place until a measured limit. The warehouse fails when it becomes a second copy of every operational table, and when the only pre-image was inside it and then transformed away. ELT plus a raw table inside the warehouse repairs the pre-image if you pay the engine's storage price and govern the raw schema. The lake repairs the pre-image at object-storage prices and then needs the lakehouse commit before curated readers are safe.

The lakehouse fails when teams keep the files and skip the commit, and when they adopt a table format without a maintenance owner. Snapshots, small files, and old metadata accumulate. The format did not remove compaction. It made compaction a new snapshot instead of a broken list. Someone still schedules it.

Use the decision lens from the course, applied here:

• Correctness. The API must see Tuesday fully replaced or not at all. The lake marker can do this only while every reader obeys it and writers do not mutate prefixes. The warehouse transaction and the lakehouse snapshot do it in the engine.
• Freshness. Lesson 47's prefix can be minutes behind the topic. The snapshot is behind the prefix by whatever the commit job's interval is. Publish both clocks. A dashboard that shows the smaller one is choosing a product it did not name.
• Scale. A year of progress events as files is the lake's job. A Monday concurrency spike of ad hoc SQL might fit a warehouse warehouse-size, or a lakehouse SQL engine pointed at the curated snapshot. Measure the concurrency and the bytes scanned before you pick. Lesson 2's warning stands: a shape that changes no lens answer is decoration.
• Reliability. Replay is the topic plus the raw prefix, until raw itself is an append-only table with its own snapshot. Restatement is a new curated version. The publication step still flips which version Monday reads, even after snapshots exist. A snapshot existing is not the same as that snapshot being the served definition.
• Security. Raw stays off the BI role. Table ACLs do not replace that. A lakehouse catalog that grants SELECT on raw to the same group as curated repeats lesson 13's email incident with better metadata.
• Cost. You pay for object storage, for the commit job, for scans, and for maintenance rewrites. You stop paying for a second full copy only if you actually stop keeping one. Two platforms serving the same grain are a cost and a correctness risk together.

Zones are layers. Metal names are optional labels

Bronze, silver, and gold show up in Databricks writing as the raw, cleaned, and serving layers. They are lesson 13's landing or raw, conformed, and curated, with a marketing vocabulary. Use them if the team already does. Do not build bronze as a new bucket that ignores s3://tlp-raw and gold as a new mart that ignores completion-v3.

Raw can remain files. That is a reasonable choice while Connect is the writer and Connect is not committing a table log. Forcing the sink to write Delta or Iceberg is a different project, with connector support you must verify for the Connect version you run. This course does not invent that connector. The commit job reads closed hours and appends or merges into a table the platform owns. The open hour stays on the streaming clock.

Curated completion should be a table before you invite a second engine. Two engines on a marker protocol means two chances to ignore the marker. Two engines on a snapshot means both pin a version or they are misconfigured. Misconfiguration is detectable. A silent list is not.

What the commit guarantees, and what it borrows from databases

Lesson 8's transaction can lock rows, run under a named isolation level, and roll back a statement that fails before commit. Readers in REPEATABLE READ keep the snapshot they started. The lakehouse analogue is: the writer commits a metadata pointer, readers that have resolved version N keep using N's file list, and a reader that refreshes resolves N+1. There is no row lock inside the Parquet file. Conflicts are detected at commit, by optimistic rules that differ per format.

Those rules are the part you must not flatten:

• Delta's default conflict story is optimistic concurrency on the files a transaction read and wrote. Two appends to a non-partitioned table can both succeed. Two jobs that rewrite the same files conflict. The loser retries. Lesson 50 states the protocol actions precisely.
• Iceberg uses optimistic concurrency on the snapshot pointer. A writer that based its work on snapshot N and finds the catalog already at N+1 retries. Lesson 51 states the metadata.
• Hudi's writer model is built around a record key and an index, with copy-on-write or merge-on-read file groups. Multi-writer needs a lock provider. Lesson 51 states that too.

A lakehouse commit does not give you a cross-table transaction for free. Publishing completion and flipping mart_publication may be two commits. The studio API still waits for the publication record, as lesson 2 required. If the engine can commit both in one place, use that. If it cannot, the publication record is the switch, and a reader that scans the fact without the switch will see a version you have not served. Snapshots make the unpublished version well-defined. They do not make it official.

Schema-on-write, for the curated table, means a commit that does not match the table schema fails. Lesson 11's Parquet file will happily store an extra field the writer stuffed in. The table format's enforcement is what rejects it, if you turn enforcement on and do not enable blind schema merge. Raw files can stay schema-on-read because the producer contract lives in the registry. Curated completion cannot. A missing attempt_number is a failed commit, not a null the studio counts as attempt zero.

Time travel is the ability to read version N or a timestamp's version after N+1 exists. It is how you audit what Monday showed, and how you recover from a bad restatement without hoping the old run prefix survived a lifecycle rule. It is not a backup strategy by itself. Vacuum and snapshot expiry delete the files old versions need. Retention of versions is a product decision: the audit window for completion-v3, aligned with delta.deletedFileRetentionDuration or Iceberg's expire_snapshots, set on purpose in lessons 50 and 51. "We have time travel" with a vacuum of zero hours is a sentence that cancels itself.

One serving system for one definition

TechLearningPro can run Databricks for the engineers who already speak Spark, and Snowflake for finance and BI concurrency. Both can see a lake. Both must not own completion-v3.

Pick a serving system per curated grain:

• PostgreSQL mart, still, when the measured volume and concurrency fit and lesson 2's limit has not been hit. A lakehouse under a small mart is a second system to operate. The raw lake can exist beside it so restatement does not return to the primary.
• Lakehouse table as the server, when Spark jobs and SQL analysts must read the same snapshot, and when restatement of a day must be a commit rather than a partition overwrite. Databricks SQL or another engine queries that table. This is lesson 52's landing.
• Warehouse table as the server, when the consumers are SQL-first, the transforms are relational, and the platform team wants the engine to own storage layout, pruning, and elastic compute. Snowflake native tables are lesson 53's landing. The lake remains the ingest record. COPY loads closed prefixes. The warehouse transaction is the commit. That architecture is a warehouse fed by a lake, which is a respectable design. Calling it a lakehouse because the files passed through S3 blurs the commit you actually have. Name the commit: the Snowflake table, or the Iceberg snapshot, not the stage.

A replica is allowed. Finance might need the lakehouse completion table copied into Snowflake for a tool that will not speak the lakehouse catalog. The replica has a freshness SLO, a definition_version, and a job id. It is not a place to "fix" completion in a personal schema. If the replica's SQL disagrees with the server, the server wins and the replica job is broken. Two gold tables with no designated server is how Monday's number depends on which tab loaded first.

Freshness clocks, written next to the dataset

Object creation time stays off the served clock. Lesson 47 already forbade quoting it as table freshness. This lesson assigns an owner to the table clock: the job that commits, on the on-call rotation that can replay it. The sink owner does not carry snapshot lag.

Governance the table does not infer

The catalog grant is the new door. USE on a schema and SELECT on curated, SELECT denied on raw, is the lesson 13 split expressed in the catalog you actually bought. A workspace that leaves raw open because "engineers need it" is a standing grant. Give engineers a role they must assume, and log it.

Column policy for prohibited fields does not move. Email never enters raw seats. Masking on a column you should not have loaded is a control after the mistake. The boundary job remains the control.

Lineage from Data Lineage and Metadata Management should name the snapshot, not only the prefix. "Came from tlp-raw" is not enough to reproduce Monday. "Came from snapshot 4412 of learning.completion, definition completion-v3, publication id ..." is enough. If the catalog you use cannot store that sentence, store it in the publication table. Do not wait for a perfect lineage product to record the id.

Quality stays at the gate. The snapshot commit after a failed uniqueness check is how bad Monday becomes durable. Lesson 17 runs, then the commit, then the publication row. Reversing that order serves the failure.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph ingest [Ingest clocks]
        topic[Kafka topic]
        prefix[Raw prefix and marker]
        dlq[Dead-letter topic]
    end

    subgraph commit [Table commit]
        job[Commit job]
        meta[Snapshot or version]
        files[Data files in object storage]
        pub[Publication row]
    end

    subgraph serve [One server]
        api[Studio API]
        sql[Analyst SQL on the server]
        replica[Optional replica with its own SLO]
    end

    topic --> prefix
    topic --> dlq
    prefix --> job
    dlq --> job
    job --> files
    job --> meta
    meta --> pub
    pub --> api
    pub --> sql
    pub --> replica

The publication row is what the API reads. The replica does not get its own definition. The dead-letter topic enters the commit job so a clean snapshot cannot hide a short hour.

mermaidCopy
sequenceDiagram
    participant Job as Restatement job
    participant Files as Object storage
    participant Log as Table metadata
    participant API as Studio API
    participant Old as Reader pinned to N

    Job->>Files: Write new data files for Tuesday
    Job->>Log: Commit version N+1 if checks pass
    Note over Old: Still reads files named by N
    API->>Log: Resolve the published version
    API->>Files: Read only that version's files
    Note over Job: A failed check commits nothing

A reader pinned to N is a feature during the commit. A reader that lists the directory is the incident this diagram removes.

Technical Explanation

Optimistic concurrency, without a format-specific myth

Two commit jobs read version N, each write their own new files, and each try to publish N+1. The catalog or the log accepts one. The other must fail and retry against the new version, or overwrite work it did not merge. Append-only jobs that add disjoint files often both succeed, because the format can take both file lists. Jobs that delete and rewrite the same day collide. Design the curated restatement as one writer per date window so the conflict is rare, and still handle the conflict. "We will not collide" without a retry is how a backfill and the morning job lose a day silently if the loser swallows the error.

Idempotency is the retry's partner. A restatement of Tuesday that is deterministic on the raw snapshot can retry safely. An append of the same batch into an unpartitioned curated table doubles completion. Lesson 13's rule, replace the keys in the window, becomes a MERGE or a partition overwrite expressed as a table operation, keyed by the grain and definition_version. The SQL lives in lessons 50 and 53. The architectural rule lives here: a retry is the same version's inputs applied again, not a second copy of the rows.

The publication row names a snapshot

Keep a small table, in the same catalog if the engine can update it in the same commit, otherwise in the operational database the API already trusts.

Columns that earn their place: dataset, definition_version, snapshot_id or table_version, event_time_window, published_at, source_marker_run_id, quality_gate_run_id. The API selects the latest publication for completion-v3 and reads that snapshot. It does not select max(snapshot). A snapshot created by compaction or by a failed experiment can be newer and must not be served. Compaction that does not change data should not move the publication row. A bad experiment should not either.

This is the lesson 2 publication row with a snapshot id added. Teams that drop it because "the table's current version is obvious" serve compaction experiments and backfills. Current and official are different words.

Engine features you may use, and features you must not assume are portable

SQL MERGE, time travel, and schema enforcement exist on Delta, Iceberg, Hudi, and Snowflake, with different syntax and different limits. Hidden partitioning is an Iceberg idea. Z-order and liquid clustering show up in Delta and Databricks writing. Micro-partition pruning is Snowflake's layout, not a file you manage. A design that requires a specific one of these names the platform. A design that requires "a snapshot, a schema, a replace of Tuesday, and a publication row" can move. Write the requirement in the second form. Name the platform in the runbook, not in the metric definition.

Cross-table foreign keys are not the lakehouse's strong suit. Lesson 10's conformed dimensions are still your job. The table will not reject an unknown course_id unless you run the lesson 17 check. Do not skip the check because the new platform feels like a database.

Streaming reads of a table, such as a job that tails new snapshots, are real features in these formats and they are easy to point at the open raw prefix by mistake. A stream on the curated table sees commits. A stream on the raw prefix sees puts. Pick the one whose clock you meant.

Cost of versions

Every restatement writes files. Every compaction writes files. Readers of old versions pin old files, so you cannot delete eagerly. The bill is storage of retained versions plus the compute of maintenance. Set retention to the audit window, not to "forever," and not to "zero so the bill drops" during an incident. Zero ends time travel and can break a long-running reader. Lesson 50's vacuum retention and lesson 51's snapshot expiry are the knobs. This lesson's decision is the window: three-day restatement plus the audit you already promised for completion, written as a number of days next to the table owner.

Small files survive the move to a table. The commit makes them a maintained backlog rather than an ambiguous directory. Maintenance is a scheduled job with the same on-call as the table, not a hope that the format compacts itself. Some platforms offer managed maintenance. Lesson 52 will name what Databricks actually runs. You still own the retention number those jobs are allowed to apply.

Security of the metadata

The log and the manifests name every file. A reader who can read the metadata and the files can read the table. A reader who can read the files and not the metadata can still parse Parquet and bypass the snapshot. Deny file-level access for human roles on the data prefix. Grant table SELECT in the catalog. Storage credentials belong to the engine's identity, which is the Unity Catalog storage credential or the Snowflake storage integration in later lessons, not a long-lived key in a notebook.

A notebook that mounts the bucket with a personal key is a bypass of the catalog. Treat it as a defect in the workspace design, not as a convenience the lakehouse has to live with.

Practical Implementation or Design Walkthrough

Problem

Move Tuesday's completion publish off the marker protocol and onto a snapshot, without creating a second definition in another platform. Leave the open progress hour on the streaming clock.

Target design

• Raw progress and purchase lines stay on lesson 48 markers until the commit job runs. The commit job reads marked, complete hours only.
• Conformed progress is a table, source grain, deduplicated on event_id.
• Curated learning.completion is a table at (learner_id, lesson_id, attempt_number) with definition_version.
• learning.publication names the snapshot Monday serves.
• The studio API reads learning.publication and then the named snapshot. It never lists s3://tlp-curated.
• Snowflake is not in this walkthrough. If finance later needs a copy, it is a replica job from the published snapshot, with a column source_snapshot_id.

Steps

1. Write the grain and the definition version at the top of the design note. If they differ from lesson 10 and lesson 13, stop and reconcile. The platform move is not a remodel.
2. Choose the serving engine for this grain. For the walkthrough, the server is a lakehouse table queried by the jobs that already run Spark, because the restatement is a large scan of progress files. Record the alternative you did not pick: keep the PostgreSQL mart if the scan still fits, per lesson 2.
3. Implement the commit as a single writer per event_date window. Inputs are the raw markers for that window and the quality results. Output is a new snapshot. On conflict, retry the whole window from the same inputs. Do not append.
4. Run lesson 17 checks before commit: uniqueness at the grain, referential match to the pinned lesson manifest, completeness against raw counts you can explain. A failure leaves the previous publication in place.
5. Insert the publication row with the new snapshot id only after the commit succeeds and the checks are the checks for that run id.
6. Point the API at the publication row. In a lab, start a long read on version N, commit N+1, and show the long read's count unchanged. Then show a refreshed official read following N+1 only after publication.
7. Schedule maintenance as a separate job that may create snapshots and must not update learning.publication.
8. Document the two clocks in the dataset page: prefix marker time, snapshot time, publication time.

A publication check you can unit-test

pythonCopy
from dataclasses import dataclass


@dataclass(frozen=True)
class Publication:
    dataset: str
    definition_version: str
    snapshot_id: int
    quality_passed: bool


def may_serve(pub: Publication, official_definition: str) -> bool:
    """Monday reads a snapshot only when it is the official definition and the gate passed."""
    if pub.dataset != "learning.completion":
        return False
    if pub.definition_version != official_definition:
        return False
    return pub.quality_passed


def clocks_are_distinct(prefix_marker_time: str, snapshot_time: str) -> bool:
    """The fields exist so a dashboard cannot bind one label to both."""
    return bool(prefix_marker_time) and bool(snapshot_time)

may_serve is the API's rule. A snapshot id alone returns false when the gate failed. Wire the real read to the engine's time-travel syntax in lesson 50 or 51. This function does not invent that syntax.

What you measure

Bytes scanned per publish. Snapshot count per day. File counts after maintenance. Lag from max marker published_at in the window to publication.published_at. That lag is the table clock. A jump means the commit job stalled while the sink stayed healthy. Page the commit owner. Do not retune flush.size as the first response. Lesson 47 already made that point. The lakehouse gives it a place to live.

Enterprise Real-World Scenarios

Two readers, one Tuesday, no snapshot

The completion backfill for a definition tweak runs in a notebook that writes new Parquet into the curated date prefix and deletes objects it considers stale. The morning job, still on the previous marker, reads a mix: some of the notebook's files, some of the old run, because the deletes and puts interleave on S3. The studio API, which only checks that a marker exists, serves the mixture for forty minutes. The count moves twice without a definition_version change. Support tickets say the course page "flickered."

The repair is the protocol in this lesson. The notebook loses write access to the data prefix. The backfill becomes a job that commits version N+1 and does not update learning.publication until the metric owner accepts the version. The API pins the published snapshot. A reader who starts on N finishes on N. The flicker was a list. The follow-up deletes the personal key that mounted the bucket. Catalog grants are the only human path. The incident review records that the marker's single-writer assumption was already false the moment analysts had puts. The lakehouse did not add bureaucracy. It removed a write path the marker could not police.

Gold in two platforms, two Mondays

Engineering serves completion-v3 from a lakehouse table. A finance project loads "the same" metric into Snowflake with a SQL model that treats a missing terminal checkpoint differently, and the BI tool points at Snowflake because that is where the other finance extracts live. Both tables are named completion. Neither publication row exists. On Monday the numbers differ by the learners the older player never marked complete, which is the exact population lesson 13 added to completion-v3. Each side trusts its engine.

The repair designates the lakehouse table as the server for this grain, because that is where the reviewed SQL and the quality gate already run. Snowflake receives a replica of the published snapshot, loaded by a job that copies rows and writes source_snapshot_id and definition_version. The finance model that reinterpreted checkpoints is retired. The BI tool's dataset description names the server and the lag. A reviewer who proposes a second transform in the replica schema is told to change completion-v3 once, at the server, and let the replica follow. The lakehouse did not forbid Snowflake. It forbade a second definition. Lesson 53 will build the replica load as COPY or as a query against a shared table, with the same rule.

Best Practices

• Require a metadata commit before any curated dataset is called a table. A location string is a lake.
• Keep lesson 13's grain and definition_version on the serving table. Change platforms without changing the metric by accident.
• Serve from a publication row that names a snapshot. Do not serve "whatever is current."
• Leave open hours on the streaming clock. Commit closed, complete hours.
• Run the lesson 17 gate before the commit that publication will name.
• Plan for one writer per restatement window, and retry the window on an optimistic conflict.
• Make retries replace the window. Do not append the same batch twice.
• Separate maintenance snapshots from published snapshots.
• Grant humans table privileges, not bucket prefixes. Keep raw off the BI role.
• Pick one server per curated grain. Replicas carry the source snapshot id and do not reshape the metric.
• Write prefix time, snapshot time, and publication time as three fields.
• Choose a table format with the criteria in the summary of this lesson, then accept that format's maintenance. Do not collect all three formats for one grain.
• Keep the PostgreSQL mart while it meets the lens. Adopt a lakehouse when the commit, the scan, or the concurrency requires it.

Common Mistakes and Anti-Patterns

Renaming the lake. The bucket is still a set of keys. Without a versioned file list, readers can mix Tuesday. The word lakehouse on the diagram does not add the list.

Medallion as a second taxonomy. Bronze that is not raw, and gold that is not completion-v3, gives you two lineages. Map names onto lesson 13. Do not fork the metric.

Serving the latest snapshot. Compaction and abandoned backfills create snapshots newer than the official one. The publication row is the switch. Latest is a debugging command.

A cross-engine transaction fantasy. Two catalogs do not commit together because both support ACID on their own tables. The publication row in the serving system is the switch the API trusts. A replica updates later, on its own SLO.

Skipping the quality gate because commits feel transactional. A commit makes the bad rows durable and time-travelable. The gate decides whether that commit is allowed to be named by publication.

File access for humans "just this once." The notebook mount bypasses schema enforcement, grants, and the snapshot. Once is how the flicker incident starts. Storage credentials belong to the engine identity.

Vacuum or expiry as a cost panic. Deleting files a retained version still names breaks audit and can break a running reader. Set retention to the audit window in advance. Expire on a schedule, not during the incident call.

Two gold transforms. The replica that "corrects" completion is a second product. Change the server. Let the copy follow.

Streaming the open prefix into the curated table to look fresh. You collapse the lesson 47 clock into the table and publish partial hours. Ingest closed hours. Let the stream serve the intra-hour product it already is.

Adopting Delta, Iceberg, and Hudi for the same grain. You triple the maintenance skills and you still need one server. Pick with the criteria below. A proof of concept can read the same raw prefix into two formats. It does not get two production publication rows.

Troubleshooting and Design Review

Symptom: The studio count changed, and definition_version did not, and nobody published.

Likely cause: the API reads the table head rather than the publication row, so a compaction rewrite with a bug, or a backfill commit, moved the number. Or a reader bypassed the table and listed files.

Investigation: publication history against table history for the window. Whether the API's query pins a snapshot id. Grants and any bucket mounts in the workspace.

Corrective action: pin the API to the publication snapshot. Revert the served id if a bad snapshot was published; the previous id should still be readable inside retention. Remove file-level grants. If retention has already deleted the previous version's files, you are in a rebuild from raw markers, and the expiry policy was shorter than the audit promise.

Symptom: Spark and the SQL engine disagree on the same snapshot id.

Likely cause: they are not actually on the same catalog pointer, or one engine caches metadata, or one is reading a path. Less often, a format feature one engine does not implement, such as a delete vector or a partition evolution, is being ignored. That class of bug belongs to lessons 50 and 51 once you know the engines differ.

Investigation: the metadata location each engine resolved. The snapshot id each one prints. A direct list of files the metadata names, compared with what each engine scanned.

Corrective action: point both engines at one catalog. Upgrade or stop using a table feature the slower engine cannot read. Do not "fix" the disagreement by averaging the counts.

Symptom: The prefix SLO is green and the publication lag is two hours.

Likely cause: the commit job is behind or failing its gate. The sink is fine.

Investigation: the job run, the quality-gate result, optimistic-conflict errors, and the last publication timestamp versus the latest marker.

Corrective action: page the table owner. Fix the gate or the conflict retry. Do not tell the sink on-call to flush more often. Publish the lag as the table clock so the next incident starts in the right queue.

Symptom: Storage cost climbs every day and queries get slower, with no new product traffic.

Likely cause: snapshots and small files retained without maintenance, or every restatement rewriting more than its window.

Investigation: snapshot count, files per snapshot, bytes added per commit, whether the job replaces a day or rewrites history.

Corrective action: schedule compaction or OPTIMIZE-style rewrites for the format you chose, then expire versions outside the audit window. Narrow the restatement to the window. A rewrite of all history belongs to a version migration with a sized budget, not to the morning job.

Symptom: Analysts report a completion figure that matches neither the API nor the replica's documented lag.

Likely cause: a personal schema re-aggregated raw, or a BI model on the replica applied its own filter.

Investigation: the query text, the tables it touches, whether definition_version appears.

Corrective action: revoke raw from that role if the query is a shadow mart. Point the BI model at the published replica. If the official number is wrong, change it once at the server and restate inside the window.

A design review asks: what is the server for this grain? What snapshot does publication name? What does a reader pinned to N see during a commit? Where is the quality gate relative to the commit? Who can read files without the catalog? What is the audit retention in days? What is the replica's rule if it disagrees? Which clock is the sink not responsible for?

Interview Questions and Answers

Q1. What is a lakehouse, in one precise sentence, and what is not one?

A lakehouse is a table whose data files live in object storage and whose readers pin a metadata version that names those files. A bucket of Parquet with a Hive table on top is a lake. A warehouse table the vendor stores and transacts, fed by files from that bucket, is a warehouse with a lake ingest. Both can be the right design. I use the word lakehouse when the commit I rely on is the open table snapshot, not the marker and not only the vendor's internal micro-partitions.

Q2. How does a lakehouse commit differ from the lesson 48 marker?

The marker is a JSON object a reader must remember to fetch, with a single-writer policy the storage system does not enforce. A lakehouse commit updates metadata so a reader that asks for version N receives a file list, and a concurrent commit either lands as N+1 or conflicts and retries. Old readers stay on N while new files are written. I still keep a publication row so "current version" and "served version" stay distinct. The log removed the mixed list. It did not decide which version is official.

Q3. When would you keep the PostgreSQL mart instead of moving completion to a lakehouse?

When the scan, the concurrency, and the restatement window still fit the engine lesson 2 already accepted, and the raw lake holds the pre-image so we are not back on the primary for history. A second platform has an operating cost. I move when a measured limit shows up: the publish no longer fits the window, analysts and Spark jobs must share a snapshot, or the marker protocol's single writer is already false. I do not move because a reference architecture drew a lakehouse.

Q4. Why is "latest snapshot" the wrong thing for the studio API to read?

Maintenance and failed experiments create snapshots that are newer than the last good publish. Latest would serve them. I store the snapshot id on a publication row after the quality gate passes, and the API reads that id. Compaction is allowed to add snapshots. It is not allowed to move the publication row.

Q5. What does optimistic concurrency not give you?

It does not give a multi-table transaction across catalogs, a row lock inside Parquet, or a promise that two writers who both rewrite Tuesday will merge your business logic for you. One commit wins. The other retries or fails. I make the restatement deterministic on its inputs and I retry the window. I still use the publication row when the fact table and the "Monday may read this" switch are not one commit.

Q6. How do bronze, silver, and gold relate to the zones you already have?

They are labels for raw or landing, conformed, and curated. I map them onto those zones, their grains, and their grants. I do not create a parallel pipeline with a new completion definition under the name gold. If a team uses the metal names in the catalog, the schema comment cites the lesson 13 zone and the definition version.

Q7. Where do dead-lettered purchase lines sit relative to the snapshot?

They block the purchase marker and they block the commit that would publish that hour, using the completeness rule from lesson 47. A snapshot of the files that happened to land is internally consistent and financially short. The gate fails. The previous publication stays. The dead-letter owner fixes the converter. The commit retries the hour. I do not publish a snapshot and attach a footnote.

Q8. What do you hand an engineer who must choose Delta, Iceberg, or Hudi after this lesson?

I hand criteria, not a winner. One serving catalog or many engines. Who the writer is, and whether they need equality merges on a key more than they need hidden partitioning. Whether Databricks is already the operating platform. Whether Snowflake native tables are the server and the open format is only the lake. Maintenance skills we actually have. Lessons 50 and 51 fill in the mechanisms. This lesson forbids choosing all three for one grain, and forbids a choice that ignores who commits.

Q9. A replica in Snowflake disagrees with the lakehouse server. Which number do you ship, and what do you edit?

I ship the server. I edit the server's SQL if the official definition is wrong, and I restate the window. I fix the replica job if it drifted. I do not patch the replica's SQL to match a finance preference. The replica row carries source_snapshot_id so the drift is measurable. Two edits in two systems is how completion-v3 forks.

Q10. Why can schema-on-read remain on the raw prefix while curated is schema-on-write?

Raw keeps the producer contract, including hours the table job has not committed, and the registry is the schema gate for producers. A curated row missing attempt_number must fail the commit so the API cannot count it as zero. The table format enforces that if we configure it to. A Parquet file alone will not. I do not enable a merge-schema option on curated to make a bad job succeed.

Q11. What three timestamps belong on the dataset page, and who is paged for each?

The prefix marker time, owned with the sink and the lake publisher. The snapshot time, owned by the commit job. The publication time, owned by whoever is allowed to flip Monday, often the same commit job after the gate. I do not page the sink because publication lags, and I do not page the table owner because an open hour has no marker. The clocks were split so the pages would split.

Hands-On Exercises

Exercise: Replace the marker with a version for one grain

Problem statement. Design the lakehouse serving path for completion-v3. Keep raw hours on markers until they are closed and complete. Show that a reader on the old version does not flicker, and that a second platform cannot invent its own completion.

Tasks.

1. Write the lake, warehouse, and lakehouse rows for Monday's completion question, using the decision lens. State which shape is the server and why the PostgreSQL option was kept or left.
2. Map raw, conformed, and curated to tables or prefixes. Mark which commits are allowed to move learning.publication.
3. Specify the publication columns, including snapshot id, definition version, and quality run id. Implement may_serve.
4. Describe a lab sequence: reader pinned to N, writer commits N+1, pinned reader unchanged, API unchanged until publication, API moved after publication. Name the failure if the API selects the latest snapshot instead.
5. Place the dead-letter completeness check in the sequence, before the commit.
6. Design a Snowflake or other replica that cannot change the metric. Include the column that proves which snapshot it copied.
7. Set an audit retention in days. State one maintenance job that must not update publication, and one cost incident if you never expire snapshots.

Expected deliverables. The decision table, the publication rule and tests, the lab sequence, the replica rule, and the retention note.

Evaluation rubric.

You pass when a reviewer can name the snapshot the API reads, the snapshot it must not read, and the system that is forbidden to redefine completion.

Lesson Summary

A lakehouse is a commit protocol on lake files. Readers pin a version. Writers publish a new version by metadata, and conflicting writers retry. Curated TechLearningPro tables keep the grain and definition_version they already had. A publication row names the snapshot Monday may read, after the quality gate, which is not always the latest snapshot. Raw hours can stay on the lesson 48 marker until they are closed and complete. Dead letters block the purchase commit. One grain has one server. A second platform is a replica with a source snapshot id. Maintenance creates versions and must not flip publication by accident. Retention matches the audit window.

The format that stores the version is the next choice. Delta Lake Architecture and Operations opens the transaction log, the vacuum retention, and the merge you will use when Delta is that format. Iceberg and Hudi follow, for teams whose catalog and writer are different.

Further Reading

• Delta Lake — the transaction-log approach lesson 50 will operate. The protocol document is the conflict behavior, not a blog summary.
• Apache Iceberg spec — snapshots and manifests. Lesson 51 walks the tree. The spec is the source of column ids and partition evolution.
• Apache Hudi — timeline, copy-on-write, and merge-on-read. Read the overview before the Spark options.
• Dimensional Modeling and Data Warehousing — the grain the serving table is not allowed to abandon.
• Relational Database Internals — transactions the lakehouse commit is often compared with too loosely.
• Data Lake Architecture — the marker protocol and its single-writer limit.
• Databricks Lakehouse Platform — one operating platform for this commit.
• Snowflake Architecture and Engineering — a warehouse server fed by the same lake, and the replica rule when it is not the server.