Data Lake Architecture

Data Engineering & Modern Data Platforms · Lesson 48 · Modern Data Platforms and Cloud

Lesson Overview

The prefixes from Real-Time Data Quality and Monitoring are now the input. s3://tlp-raw/learning/progress/ holds hourly Parquet from the Connect sink on learning.progress. s3://tlp-raw/commerce/purchase_line/ holds purchase lines, with skips on commerce.purchase_line.dlq. A promotion hour is still open. Forty objects exist, the sink will write fifteen more, and an analyst has already pointed a notebook at the prefix and published a revenue figure. The objects that exist are real. The hour is not a version. Amazon S3's consistency model makes a subsequent read of one object see the put that finished. It does not make a prefix of fifty-five objects appear as one commit.

Modern Data Architecture Overview named the lake as files in object storage, with schema applied when a reader decides to apply it, and warned that a successful put is not an atomic replacement of Tuesday. ETL and ELT Architecture already split landing, raw, conformed, and curated so that completion-v3 could be restated from retained fields. Hadoop Ecosystem and HDFS showed why a rename inside one HDFS namespace is a metadata operation and why the same rename on an object store is a copy. This lesson is the layout those warnings become: zones with owners, hive-style partitions that match the contracts, a catalog that remembers columns, a file-size budget, and a publication marker that names which prefix a reader may scan.

The marker is the lake's stand-in for the mart_publication row in lesson 2. It records a run id, an object count, a max event time, and a prefix. It is not a snapshot. Two writers can still race. A reader that ignores the marker and lists the bucket will see the open hour. Data Lakehouse Architecture replaces the marker with a table commit. Delta Lake Architecture and Operations and Apache Iceberg and Hudi Fundamentals are the formats that commit. This lesson refuses to pretend a folder is already one of them.

Learning Objectives

By the end of this lesson you will be able to:

1. Place landing, raw, conformed, and curated on object storage with an owner, a format, and a retention for each TechLearningPro dataset.
2. Explain why strong consistency of a single object put does not make a multi-object prefix a commit.
3. Choose hive-style partitions from the contract's time field, and refuse a partition on learner_id.
4. Separate a metastore or Glue catalog entry from a transaction log.
5. Size files for a scan, and treat Connect flush.size output as a compaction input on closed hours only.
6. Publish a marker that names the prefix, the object count, and the event-time high-water, and block readers that skip it.
7. Keep raw retention aligned with Kafka retention so a topic expiry does not delete the only replay copy.
8. Hand the lakehouse a closed prefix, a schema, and an explicit statement that object-listing time is not table freshness.

Prerequisites

• Lesson 2: warehouse, lake, and lakehouse as shapes. The publication row is the consumer contract. This lesson builds the lake shape's files.
• Lesson 10: grain. Curated completion stays (learner_id, lesson_id, attempt_number). The lake does not get to invent a second grain because the files are wide.
• Lesson 11: Parquet for scans, JSON or CSV as received, row groups and the small-file warning. This lesson places those files.
• Lesson 13: landing, raw, conformed, curated, and ELT so a definition change can restate from retained fields. Zone names in this lesson are those names.
• Lesson 17: fail, quarantine, and warn. The lake marker is where a completeness failure stops a reader. It does not replace the curated gate.
• Lesson 25: HDFS rename versus object-store copy, and namespace cost of small files. The same cost shows up as listing time on S3.
• Lesson 42: the sink paths, dead-letter topics, and flush.size. Retention on the topic must cover the sink outage you will replay.
• Lesson 47: the handoff table. Prefix freshness and table freshness are different clocks. This lesson owns the prefix clock's reader rule.

Detailed Core Concepts

What a lake is, on this platform

A data lake is a durable store of files, addressed by keys, that keeps data in the format the producer or the landing job wrote, at a cost that makes retention of the pre-image affordable. TechLearningPro's lake is Amazon S3 because lessons 42 and 47 already named s3://tlp-raw/.... Azure Data Lake Storage and Google Cloud Storage are the same shape with different identity systems. The lessons that follow on each cloud fill in the identity. The architecture here is the keys, the zones, and the reader rule.

James Dixon's picture, a lake you draw from rather than a bottled mart, is the history. The engineering content is narrower. The lake holds bytes a later definition might need. It does not answer Monday's completion question by itself. The curated mart still does, at the grain lesson 10 fixed. The lake is why that mart can be rebuilt when completion-v3 changes, which is the failure lesson 13 opened with.

A lake becomes a swamp when keys have no owner, no partition convention, no schema version, and no check that a prohibited column stayed out. Analysts then disagree because they parsed different hours with different null rules. The repair is the layout in this lesson, not a new bucket.

Zones, with the paths this course already uses

Lesson 13's four zones stay. Object storage gives each one a prefix. A prefix is not a security boundary by itself. Identity policies make it one.

Landing is private because lesson 13's seat file can contain email until the boundary job rejects it. A catalog table on landing with the BI role granted SELECT recreates that incident. Raw progress is still personal data in combination: a learner id and a timeline. The transform role may read it. The BI role reads curated. Break-glass on raw is time-limited and logged.

The Connect sink writes raw, not landing. Lesson 42 already converted the topic into Parquet and sent failures to the dead-letter topic. Do not insert a second copy of every object into landing "for safety." Two prefixes without a rule for which one is authoritative will diverge during a replay.

Conformed and curated, in this lesson, are still files. A Spark job can write them. Readers may scan them only through the marker defined below. They are not yet Delta, Iceberg, or Hudi tables. Calling the prefix a table in a slide does not create a transaction log.

One object is atomic. A prefix is not

S3 documents strong read-after-write consistency for puts of new objects and for overwrites and deletes. A GET or a HEAD after a successful PUT returns that object. A list after a successful PUT includes it. That ended an older era in which a reader might miss an object it had just written. It did not add a multi-key transaction.

A Connect task finishing part-00012.snappy.parquet makes that key visible. A task still uploading part-00013 has not. A list of event_date=2026-11-10/hour=14/ returns a prefix in progress. Summing it as the hour's revenue is a partial read. Consistency made each key honest. The hour is a set of keys, and the set is not a commit.

Multipart upload has a second gap. An incomplete upload is not a readable object, and it still accrues storage until it is aborted or a lifecycle rule aborts it. A failed writer that does not abort leaves parts. The lake's lifecycle configuration includes an abort of incomplete multipart uploads. That rule is operational hygiene. It is not the publication protocol.

HDFS, from lesson 25, can rename a temporary directory into the published path inside one namespace, and readers see the destination appear with its entries. Object storage has no prefix rename. Copying each key and deleting the source is a loop. A reader listing during the loop sees a mixture. Hadoop's S3A connector documents committers (directory, partitioned, magic) because of this gap. Use the committer your runtime's S3A documentation describes if Spark is writing the curated files. Then still publish the marker. A committer that makes task output appear under the final keys does not tell a human which hour is closed.

Partitions are a contract with the query, not a dump of every column

Hive-style partitioning encodes a column in the key:

textCopy
s3://tlp-raw/learning/progress/event_date=2026-11-10/hour=14/part-00012.snappy.parquet
s3://tlp-raw/commerce/purchase_line/event_date=2026-11-10/hour=14/part-00003.snappy.parquet

event_date is the UTC date of event_time from the lesson 47 contract, not the wall clock of the put. A late event from Tuesday that arrives Wednesday lands in Tuesday's partition if the sink's partitioner uses the record timestamp, which is the setting lesson 42's rotate.interval.ms discussion assumed. Document the choice on the connector. A sink that partitions by processing time will put Tuesday's event in Wednesday's prefix, and the morning restatement will miss it while a "today" dashboard counts it.

hour exists because the tile and the promotion check care about a slice smaller than a day, and because compaction wants a closed unit smaller than a full day of progress. Hour is optional on curated completion, which lesson 13 restates by day. Match the partition to the replace unit. A curated table partitioned by hour when the job replaces a day forces the job to touch twenty-four prefixes for one definition change.

Do not partition by learner_id, order_id, or event_id. Cardinality explodes the listing, defeats pruning for the scans you actually run (a day, a course, a promotion window), and writes identifiers into keys that show up in access logs and in support tickets. High-cardinality filters belong inside Parquet column statistics, which lesson 11 already covered, or inside a table-format min/max index in later lessons.

Over-partitioning is the same failure at a smaller grain. An hour that contains one small file, across a quiet night, is a listing tax. Keep hour where the live prefix needs it. Compact closed hours into fewer files under the same partition, or drop hour when you write conformed. Write the rule down so the next job does not add minute=.

The catalog remembers schema. It does not commit data

A Hive metastore, or the AWS Glue Data Catalog that speaks the same table shape, maps a database and table name to a location, a column list, a partition list, and a storage format. Spark and Athena and Trino can then say learning.progress_raw instead of a raw path. That is worth having. It is a pointer and a schema cache.

sqlCopy
CREATE EXTERNAL TABLE learning.progress_raw (
  event_id        string,
  learner_id      string,
  course_id       string,
  lesson_id       string,
  event_time      timestamp,
  event_type      string
)
PARTITIONED BY (event_date string, hour string)
STORED AS PARQUET
LOCATION 's3://tlp-raw/learning/progress/';

MSCK REPAIR TABLE walks the prefix and adds partitions it finds. On a large raw tree it is a listing job, and it will register an hour that is still open. Prefer ALTER TABLE ... ADD PARTITION from the same job that writes the publication marker, for hours the marker names as closed. A partition the metastore knows is a pruning hint. It is not a promise that the files under it form one snapshot.

Glue crawlers sample objects and infer a schema. Inference on raw JSON will happily promote a field, change a type, or add a column from one bad file. The contract from Schema Registry and Event Contracts is the schema. The crawler is a convenience for a dataset nobody has contracted. Do not point a crawler at tlp-raw and call the result the progress contract.

Catalog statistics, when a engine computes them, help a cost-based optimizer. They go stale the moment a new hour lands. Recompute them on closed partitions after the marker, or accept a bad plan. Stale stats are a performance defect. They are not a correctness mechanism.

File size is a reader cost and a listing cost

Lesson 11 and lesson 25 already set the shape. A Parquet file on the order of a few hundred megabytes gives a scan engine a split it can open without spending the job on listings. A file of a few hundred kilobytes, repeated hundreds of thousands of times, spends the morning in LIST calls. Spark's default split size and Parquet's row-group default sit near 128 MB. Treat that as a planning anchor, then measure open time and bytes scanned on the completion job. There is no universal optimum to copy from a blog.

Lesson 42's flush.size is a record count. A small flush during enrollment week writes a large number of objects per hour. That is acceptable for the sink, which must not buffer an unbounded hour in memory. It is not acceptable as the layout the restatement scans forever. A compaction job rewrites closed hours into larger Parquet files and publishes a new marker pointing at the compacted prefix. It does not rewrite the current hour. It does not delete the uncompacted keys until the new marker is in place and any reader pinned to the old marker has finished. Lifecycle can expire the old run prefix after the audit window.

Compaction is a batch job with an owner. It is not a quality failure by itself. Lesson 47 said object counts are a cost metric unless readers time out. This lesson is where that metric gets a threshold: objects per closed hour, median bytes per object, and a page to the compaction owner when the restatement's listing time breaks the morning window. The page is not a page to the Connect on-call unless the sink is still writing the hour you are trying to compact.

The publication marker is the reader rule

Until a table format exists, readers need a record of which prefix is closed. The marker is a small JSON object, or a row in the operational database the orchestrator already uses. JSON in the lake keeps the rule next to the bytes. A database row is easier to lock. Pick one. This lesson uses an object so the example has no extra service, and it says so when the race matters.

A closed hour of raw progress looks like this:

jsonCopy
{
  "dataset": "learning.progress",
  "zone": "raw",
  "event_date": "2026-11-10",
  "hour": 13,
  "prefix": "s3://tlp-raw/learning/progress/event_date=2026-11-10/hour=13/run_id=20261110T140500Z/",
  "object_count": 12,
  "bytes": 1843201100,
  "max_event_time": "2026-11-10T13:59:58Z",
  "run_id": "20261110T140500Z",
  "definition_version": "progress-raw-1",
  "published_at": "2026-11-10T14:06:11Z"
}

The sink writes objects under run_id=<id>/ inside the hour, or the compaction job does. The marker is written after the writer lists that prefix and the count matches the writer's own record. Readers of "closed raw" resolve the marker and scan prefix. They do not list hour=13/ and take whatever keys exist. An open hour has no marker. The promotion dashboard that needs the open hour is a different product, with the streaming SLO from lesson 47, not a lake scan.

The marker's object_count must equal a list of the prefix at publish time. A mismatch fails the publish. A reader that recounts and disagrees with the marker stops and pages the publisher. Silence would mean the marker and the bytes have diverged.

The marker is a reader rule with three limits. It is not snapshot isolation: deny the job role DeleteObject on a published run, and let a maintenance role delete only a prefix the orchestrator has marked superseded after the audit window. Two markers for one hour means two writers; the rule is one writer per dataset hour, coordinated by the orchestrator from Apache Airflow Architecture and DAGs. A cached list goes stale if that immutability is broken. The lakehouse log exists for teams that will not hold this discipline. Curated completion uses the same shape at day grain, with definition_version from lesson 13. Replacing Tuesday writes a new run prefix and a new marker. The previous marker stays for the audit window.

Retention meets the Kafka contract

If the topic ages out, the object is the surviving copy. Raw retention is topic retention from Kafka Topics, Partitions, and Replication, plus the replay margin, plus lesson 13's audit window. Curated follows the metric owner's promise. A bucket-wide lifecycle expires Tuesday's mart together with landing.

Identity, encryption, and what must not be in the key

Separate IAM roles: the Connect sink can PutObject on the raw prefixes it owns and cannot DeleteObject on a published run, cannot read tlp-landing seat files, and cannot read the identity store. The transform role can read raw and write new conformed and curated run prefixes. The BI role can read curated markers and curated prefixes and cannot list raw. The maintenance role can delete only under a prefix the orchestrator has marked superseded.

A KMS key per zone keeps encryption aligned with the roles. Access logs record keys, which is why learner_id stays out of the path: the body is raw-role data, and the logs are wider. Block public access on these buckets. A signed URL for a support export is time-limited and owned.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph writers [Writers]
        connect[Connect sink]
        partner[Partner file drop]
        compact[Compaction and conform job]
        curatedJob[Completion publish job]
    end

    subgraph zones [Prefixes]
        landing[tlp-landing]
        raw[tlp-raw closed run prefix]
        conformed[tlp-conformed]
        curated[tlp-curated]
    end

    subgraph readrule [Reader rule]
        marker[Publication marker]
        catalog[Metastore or Glue table]
        bi[BI and studio API]
        transform[Transform role]
    end

    partner --> landing
    connect --> raw
    raw --> compact
    compact --> conformed
    conformed --> curatedJob
    curatedJob --> curated
    compact --> marker
    curatedJob --> marker
    marker --> transform
    marker --> bi
    catalog --> transform
    landing -.-> transform

The dotted edge is the refusal. The transform does not read landing. BI does not read raw. The catalog is a name for a location. The marker is what makes a location closed.

mermaidCopy
sequenceDiagram
    participant Sink as Connect sink
    participant Raw as Run prefix
    participant Pub as Marker writer
    participant Reader as Restatement job

    Sink->>Raw: PUT part files for hour 13
    Note over Raw: Hour 14 has no marker yet
    Sink->>Pub: Writer record of object count
    Pub->>Raw: LIST run prefix
    alt Count matches and hour is closed
        Pub->>Pub: PUT marker object
        Reader->>Pub: GET marker
        Reader->>Raw: Scan only that prefix
    else Count mismatches or hour still open
        Pub->>Pub: Fail the publish
        Note over Reader: No marker, no scan
    end

Hour 14 stays on the streaming clock from lesson 47. The lake clock starts when the marker exists.

Technical Explanation

How a reader is supposed to prune

A Spark read of the table, with partitions registered for closed hours only:

pythonCopy
# Spark 3.5. Partition columns come from the path when the
# DataFrame reader infers them, or from the metastore when
# you query the external table. Filter on the partition column
# so the listing stays inside one day.
spark.read.table("learning.progress_raw").where("event_date = '2026-11-10' AND hour = '13'")

If the metastore still has an open hour registered, the filter returns a partial hour and looks successful. Registration is part of publish, not a crawler loop. spark.sql.sources.partitionOverwriteMode matters when a job writes curated back as files. static, the default, can replace every partition the table knows when you overwrite. dynamic replaces only partitions present in the write. Dynamic mode is the one a daily restatement wants, and it is still not a snapshot other engines observe atomically on S3. The marker remains the publish signal. Set the mode in the job's configuration and in the review checklist so a default does not wipe Monday while rewriting Tuesday.

A query that filters event_time but not event_date may still list every partition and then apply the timestamp inside files, depending on the engine. Predicate pushdown into Parquet helps inside a file. It does not invent partition pruning if the path was never skipped. Put the partition filter in the job. Test the plan, or the listing count, on a day with more than one partition present.

The marker check is ordinary Python

The publisher and the reader can share a small function. It does not call S3. The list result is an argument, which keeps the test hermetic. Versions: this is the standard library on Python 3.11, the same baseline as the batch jobs in module 2.

pythonCopy
from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class Marker:
    dataset: str
    prefix: str
    object_count: int
    max_event_time: datetime
    run_id: str
    definition_version: str


def publication_ok(marker: Marker, listed_keys: list[str]) -> bool:
    """A closed prefix is readable when the marker count matches the list.

    An empty list is not a closed hour. A mismatch means the writer
    finished a different set of keys than the one just listed.
    """
    if marker.object_count <= 0:
        return False
    if len(listed_keys) != marker.object_count:
        return False
    return all(key.startswith(marker.prefix) for key in listed_keys)


def partition_key_allowed(key: str) -> bool:
    """Hive-style date and hour only. Identifiers do not belong in the key."""
    banned = ("learner_id=", "email=", "order_id=", "event_id=")
    return not any(piece in key for piece in banned)

publication_ok fails closed. A reader that catches the false and scans the parent prefix has undone the function. The job's control flow stops.

Wire lesson 17's purchase completeness beside this check. object_count matching does not prove every line_seq arrived. Dead-lettered lines are absent from the prefix on purpose. Completeness remains max(line_seq) against landed identities plus dead-letter identities, from lesson 47. The lake check adds "the objects we claim are the objects that are there." Both must pass before conformed purchase lines are eligible to build curated revenue. A green marker with a dead-letter hole is the short hour again, now with extra JSON.

Lifecycle and incomplete uploads

Bucket lifecycle is the documented S3 mechanism for expiration and for aborting incomplete multipart uploads. Express it per prefix:

• tlp-landing/: expire objects after the short window, abort incomplete uploads after a day.
• tlp-raw/.../run_id=: do not expire a run the current marker or the audit window still names. A superseded run expires after the audit window.
• tlp-curated/: expire only runs the metric owner has dropped. A single rule of "delete after 30 days" on the bucket is the anti-pattern.

Object Lock can make a prefix immutable for a compliance window. Use it when the audit promise is a legal one. It will also prevent you from deleting a bad publish until the lock ends, so the marker must be able to point at a corrected run while the bad run sits locked. Immutability of keys is compatible with a new marker. It is not compatible with overwriting the same key in place.

Cost shape

Storage of Parquet for a year of progress is usually the small line. LIST and GET request charges, and the compute that opens tiny files, dominate a bad layout. Catalog crawlers that run every hour add listing traffic of their own. Compaction spends compute once so every later scan spends less. Measure bytes scanned by the completion job and object counts per hour. Those two numbers tell you whether to compact, whether to drop the hour partition on conformed, and whether flush.size should change. Changing flush.size without a closed-hour compaction plan only moves the problem to the sink's memory.

Keep one authoritative prefix per closed hour, named by the marker. A second copy needs a restore procedure and an expiry. A Hive external table, a Spark read, and an Athena query are names for a scan. None of them is a multi-file commit. Data Lakehouse Architecture replaces the marker when you need concurrent writers, a MERGE on a key, or a reader that stays on a snapshot while compaction runs. Renaming the bucket does not create those properties.

Practical Implementation or Design Walkthrough

Problem

Publish one closed hour of progress and one closed hour of purchase lines so the morning restatement can scan them, and keep the open hour off that scan. Show the marker, the partition layout, and the checks that fail a partial prefix.

Sources and targets

Progress source is the Connect prefix from lesson 42, contract fields from the event: event_id, learner_id, course_id, lesson_id, event_time, event_type. Purchase source is order_id, line_id, course_id, amount_cents, event_time, line_seq. Targets are run prefixes plus markers. Conformed dedupe and curated completion-v3 stay the lesson 13 SQL, reading only marked prefixes.

Sample progress row, one line of a Parquet file, shown as a record so the key design is obvious:

jsonCopy
{"event_id":"evt_91","learner_id":"lrn_1042","course_id":"course_17","lesson_id":"lesson_48","event_time":"2026-11-10T13:14:02Z","event_type":"checkpoint"}

The key for that file does not contain lrn_1042.

Steps

1. Confirm the sink writes under event_date and hour derived from event_time in UTC. Confirm learner_id is a column, not a path segment. partition_key_allowed is the regression check in CI on any new sink config.
2. Treat the current hour as open. No marker. The streaming board from lesson 47 remains the freshness signal for that hour.
3. When the hour closes, the compaction job lists the sink keys for that hour, writes larger Parquet files to .../hour=13/run_id=<id>/, and records the count it wrote.
4. The publisher lists the run prefix. publication_ok must be true. Then it puts the marker object at a known key, for example s3://tlp-raw/learning/progress/_markers/event_date=2026-11-10/hour=13.json. The marker key lives beside the data so a reader can find it. It is not inside the run prefix, or a list of the run prefix would count it as data.
5. The metastore adds the partition only after the marker put succeeds. The location registered is the run prefix, not the parent hour.
6. The purchase path does the same, then the lesson 17 completeness check includes the dead-letter topic. A failed completeness check does not publish the purchase marker.
7. The restatement reads markers for the window, scans those prefixes, and writes conformed and curated run prefixes with their own markers and definition_version.
8. A retry uses a new run_id if the previous publish did not put a marker. If a marker exists and the count still matches, the retry is a no-op. Overwriting a published run prefix is forbidden to the job role.

Validation

Unit-test publication_ok with a matching list, a short list, an empty list, and a key outside the prefix. Unit-test partition_key_allowed with a learner_id= key. In a lab bucket, write three objects, publish a marker with object_count 3, and show the reader function accepts them. Delete no objects. Write a fourth object into the same prefix and show the next check fails. That failure is the immutability alarm. The corrective action is a new run prefix, not an edited marker that quietly updates the count.

Expected lab result: the open hour is absent from the marker store; the closed hour's scan sees three objects; a mutated prefix fails the next reader before it builds completion.

Operations

Airflow, or the orchestrator you already run from lesson 20, is the single writer. The task logs the run_id, the count, and the marker key. It does not log the Parquet body. Alert on publish failure, on count mismatch, and on objects-per-hour above the budget you set from a normal week. Do not alert on an open hour lacking a marker. That absence is the design.

Enterprise Real-World Scenarios

The promotion hour that was still open

During a course promotion, finance asks for purchase revenue "from the lake" at 14:20 UTC. Hour 14 has no marker. A notebook lists s3://tlp-raw/commerce/purchase_line/event_date=2026-11-10/hour=14/ and sums amount_cents across the twelve objects already visible. The sink is healthy, lag is inside the lesson 47 SLO, and it will write more objects before 15:00. The notebook is short. It is also missing every line the dead-letter topic holds from a converter failure earlier in the hour. Two holes, one sum, and a figure that reaches a channel.

The repair is a reader rule, enforced by the absence of a marker and by a role that cannot list the parent hour without the transform job. Finance gets either the last closed hour, with completeness that includes the dead-letter count, or the streaming aggregate on commerce.course_revenue from lesson 44, whose freshness is a different contract. They do not get an ad hoc list. The follow-up is a short note on the promotion runbook: intra-hour revenue is the stream, closed-hour revenue is the marker, and the morning books are the curated publish after lesson 17's gate. The lake did not fail to store the lines. A scan without a version did.

Enrollment week fills the prefix with tiny files

flush.size was tuned for a quiet week. Term start multiplies progress events. Hour 16 produces tens of thousands of small Parquet objects. The Connect sink stays RUNNING. Lesson 47's freshness SLO holds because objects are landing. The morning Spark restatement lists hour after hour and misses the window that publishes completion-v3. Someone scales the driver. The list becomes faster and the bill goes up. The next morning is the same shape with a larger driver.

The compaction owner, who did not exist until this incident, rewrites closed hours into a handful of files each and points the marker at the new run prefix. The restatement scans marked prefixes only. Listing time returns inside the window. flush.size stays small enough that the sink does not hold a huge buffer. The file-count chart, which lesson 47 asked Module 5 to expect, pages the compaction job when a closed hour exceeds the budget. It does not page the sink owner for doing what the connector was configured to do. Scaling the driver is how you pay twice for a layout problem.

A second effect shows up in the metastore. A crawler that registered every minute-level prefix someone added "temporarily" during the incident now has a partition list the planner prints instead of reading data. The cleanup drops those partitions, restores date and hour, and retires the crawler on this dataset. The contract columns go back to being declared in the CREATE TABLE, reviewed like code.

Best Practices

• Give each zone a prefix, a writer role, a reader role, and a retention number that cites the lesson that required it.
• Partition raw progress and purchase lines by UTC event_date and, where the replace unit needs it, hour. Keep identifiers in columns.
• Register metastore partitions from the publish job for closed hours only. Keep crawlers off contracted datasets.
• Write run prefixes that published jobs do not mutate. Put the marker outside the data prefix so a list of data files stays a list of data files.
• Fail the publish when the listed count differs from the writer's count. Fail the reader when the marker is missing.
• Compact closed hours. Leave the open hour to the sink and to the streaming SLO.
• Set lifecycle per prefix. Abort incomplete multipart uploads. Do not inherit landing's expiry onto raw or curated.
• Keep raw retention longer than topic retention by a margin you can replay.
• Deny the BI role on raw and on landing. Deny the sink role delete on published runs.
• Put definition_version on curated markers, one serving version per consumer, as lesson 13 required.
• Log marker keys and counts. Do not log Parquet rows into the platform log.
• Hand lakehouse design the closed prefix, the schema, the dead-letter rule, and the sentence that a list is not a snapshot.

Common Mistakes and Anti-Patterns

Pointing a query at a parent prefix. The list is a directory of whatever has finished uploading. The hour looks queryable because Parquet parses. The total is partial. Readers resolve a marker.

Partitioning by learner or order id. Listings explode, access logs gain identifiers, and day-scoped jobs stop pruning. The column belongs in the file. Lesson 11's column statistics are the selective filter.

A Glue crawler as the contract. Inference adds a column from one bad object and the restatement sees a schema the registry never accepted. Declare the table. Add partitions when you publish.

One lifecycle rule for the bucket. Landing's seven days deletes the only copy of progress after the topic has expired, or it deletes curated history the metric owner still serves. Prefix rules differ because the promises differ.

Registering the open hour because the files exist. The metastore then helps every engine skip straight to the partial data. Registration follows the marker.

Compacting the hour the sink is still writing. The compaction list and the sink puts interleave. The marker count flaps. Compact an hour only after the sink has moved on and the streaming SLO says the hour's objects have landed, plus the dead-letter check for purchase lines.

Treating S3 consistency as a multi-file commit. Each key is honest. The set of keys is a set. The marker, and later the table log, is what names the set.

Putting email or another prohibited field in raw because the lake is "just storage." Lesson 13's boundary stands. The lake retains the contracted pre-image. It does not retain columns the boundary was built to drop.

A second raw prefix "for backup" that jobs read at random. Replay needs one authoritative key space. A backup is a copy with a restore procedure, not a prefix analysts discover first.

Static partition overwrite on a curated file table. The default Spark mode can remove partitions you did not intend to replace. Use dynamic overwrite for a restatement, and still publish a marker. Better: stop using file overwrite as the commit once lesson 49's table exists.

Troubleshooting and Design Review

Symptom: The restatement's completion count is lower than the streaming tile for the same hour, and the hour's marker exists.

Likely cause: the marker was published before the sink finished, or dead-lettered purchase lines were ignored on a revenue comparison, or the tile includes an open window the lake correctly excluded. Confirm which product you are comparing. The tile's clock is lesson 47. The lake's clock is the marker.

Investigation: marker published_at against the sink's last put in that hour. object_count against a fresh list. For purchases, the completeness function including the dead-letter topic. For progress, whether the tile's hour is closed in event time or still inside grace.

Corrective action: if the prefix mutated after publish, cut a new run from the topic while retention allows, or from an immutable copy if you kept one, and publish a new marker. Revoke the writer's ability to put into an existing run prefix. Do not "fix" the count in the marker to match a mutated list without a new run_id.

Symptom: Morning jobs spend most of their time before the first task runs, and the driver log is a long list of keys.

Likely cause: small files, or a partition scheme that added a high-cardinality column, or a crawler-registered tree of empty prefixes.

Investigation: objects per hour and median object bytes for the dates in the window. A sample of keys. The metastore partition count for the table.

Corrective action: compact closed hours. Drop illegal partitions. Stop the crawler. Do not raise driver memory as the only change. Add the object-count check to the publish so the next enrollment week fails the compaction SLO in the afternoon, not the completion SLO the next morning.

Symptom: Tuesday's curated completion is gone, and nobody ran a restatement.

Likely cause: a bucket-level lifecycle rule, or a static partition overwrite, or a maintenance role that deleted the prefix the current marker still names.

Investigation: marker for that date, then HEAD on a key inside its prefix. Lifecycle configuration on the bucket. The overwrite mode of the last curated job. CloudTrail or access logs for DeleteObject if you enabled them.

Corrective action: restore from the superseded run if it has not expired, and point the marker back. Narrow the lifecycle. Split roles so the job that writes cannot delete a published prefix. If no copy remains, the rebuild path is topic replay when retention allows, else the incident is data loss and the retention design was the defect.

Symptom: Two markers exist for hour=13 and two jobs each claim they published.

Likely cause: the orchestrator allowed overlapping runs, or a manual notebook published beside the DAG.

Investigation: both markers' run_id and published_at. The DAG's task instance history. Whether both prefixes are immutable and complete.

Corrective action: choose the marker whose count matches a list and whose completeness check passes. Disable the other prefix from readers by leaving it unreferenced. Enforce a single active DAG run per dataset hour. A lake does not have a compare-and-swap on the marker unless you add one in the database. If this race is recurring, that is a requirement for the lakehouse commit in the next lesson, not for a more elaborate JSON file.

Symptom: A reader with the raw role can see partner CSV headers that include email.

Likely cause: landing and raw share a bucket policy, or a crawler created a table over tlp-landing in a database the transform role can browse.

Investigation: the table's LOCATION. The role's allowed prefixes. A sample header, handled as sensitive: confirm the column name, do not paste the file into chat.

Corrective action: drop the catalog entry. Restrict the role. Re-run the boundary job from lesson 13. Expire the landing objects on their short clock. The curated seat fact should already lack the column. If it does not, stop serving it.

A design review asks: for each dataset, what is the prefix, the partition, the writer role, the reader role, the retention, and the marker key? Which hours are allowed to have no marker? Where does the dead-letter count enter? What deletes a published object, and which role is denied? What happens on retry after a failed marker put? Which number is a list, and which number will the lakehouse be forbidden to quote as a snapshot?

Interview Questions and Answers

Q1. What is a data lake, and what problem does it solve for TechLearningPro that the curated mart does not?

A lake is a durable collection of files, usually in object storage, that keeps data in the format we landed, cheaply enough to retain the pre-image. The curated mart answers a grain we already agreed, such as completion at (learner_id, lesson_id, attempt_number). The lake is why we can restate that mart when the definition changes, which is the lesson 13 failure when checkpoints were dropped before they reached an analytical store. The lake does not replace the mart. A scan of raw progress is not completion-v3.

Q2. S3 is strongly consistent. Why can a reader still see a partial hour?

Consistency applies to each key. After a put succeeds, a later read of that key sees it, and a list includes it. An hour is many keys. Some puts have succeeded and some have not. A list in the middle is an honest list of a set that is not the set we intend to publish. I publish a marker only after the writer's count matches a list of a finished run prefix, and readers scan that prefix. I do not treat the parent hour as a version.

Q3. Why is an HDFS directory rename a poor mental model for publishing a lake hour?

Inside one HDFS namespace, rename is a metadata operation. Readers see the destination path with its entries. Object storage does not rename a prefix. A copy loop is visible mid-way. S3A committers exist to make Spark task commits less likely to expose partial output, and they still do not name which hour is closed for every engine. I keep an explicit marker, and I move to a table log when one marker writer is no longer enough.

Q4. How do you choose partitions for progress events?

I partition by the UTC date of event_time, and by hour when the sink and the compaction unit need an hour. Those are the filters the restatement and the freshness checks use, so pruning avoids listing the year. I do not partition by learner_id. That cardinality creates a small-file and listing incident and copies identifiers into access logs. Selective access by learner is a column filter inside Parquet, or a later table-format stat, and it is still subject to the raw-role grant.

Q5. What does a Hive or Glue catalog provide, and what does it refuse to provide?

It provides a table name, a schema, a location, and partition entries so engines can prune. It can store statistics that go stale. It does not make the files under a partition one atomic snapshot. MSCK REPAIR will register whatever prefixes exist, including an open hour. I add partitions from the publish job after the marker succeeds. I do not let a crawler infer the progress contract from samples.

Q6. How should raw retention relate to Kafka retention?

The object is the surviving copy once the topic expires, which lesson 42 already said. Raw retention has to cover topic retention plus the replay margin plus the audit window we need for restatement. I set lifecycle on the raw prefix to that sum. I do not copy landing's short expiry onto raw. If I cannot state the three numbers, I am not ready to attach a lifecycle rule.

Q7. A marker's object_count matches the list, and finance still under-counts purchases. What did the lake check miss?

Lines on commerce.purchase_line.dlq never became objects. The prefix is internally consistent and incomplete. Lesson 47's completeness function counts landed identities and dead-letter identities against max(line_seq). I run that check before I publish the purchase marker. A matched object count is necessary and it does not speak for records the sink quarantined.

Q8. Why must the publication marker live outside the run prefix?

A list that validates object_count should count data files. A marker object inside the prefix changes the count, gets scanned as if it were Parquet, or both. I write .../_markers/...json beside the data and point prefix at the run directory. The same split keeps a lifecycle rule from expiring markers and data on one clumsy filter, once I attach rules deliberately.

Q9. What is the small-file failure during enrollment week, and what do you scale?

The sink flushes many small objects because flush.size is a record count chosen to bound memory. The morning job then lists them and misses the publish window. I scale the layout: compact closed hours into larger files and scan the marked prefix. I keep the sink's flush unless the sink itself is failing. A larger Spark driver lists faster and does not change next term's object count.

Q10. When is the marker protocol no longer enough?

When two writers must publish the same dataset without an orchestrator lock, when readers must stay on a snapshot while compaction rewrites files, when we need MERGE on (order_id, line_id) as a transaction, or when time travel has to be a version number rather than "point at an old run prefix and hope nobody deleted it." Those are lakehouse requirements. I hand over the closed prefix, the schema, the dead-letter rule, and the clocks. I do not implement them by adding fields to the JSON until the JSON is a bad transaction log.

Hands-On Exercises

Exercise: Close one hour without closing the open one

Problem statement. Design the raw lake layout for progress and purchase lines. Publish a marker for a closed hour. Prove an open hour and a mutated prefix do not scan as success.

Tasks.

1. Draw the four zones with paths, roles, and retention. Cite the Kafka retention number you assume, and show raw retention is longer.
2. Specify hive-style keys for both datasets. Include one illegal key, and show partition_key_allowed rejects it.
3. Implement Marker and publication_ok. Test a match, a short list, an empty list, and a key that escapes the prefix.
4. Write the marker JSON for 2026-11-10 hour 13 and the metastore ADD PARTITION you would run only after publication_ok returns true. State the location clause.
5. State how purchase completeness from lesson 47 gates the purchase marker, in addition to the object count.
6. Describe compaction: what it reads, what it writes, what it will not touch while the sink is inside the hour.
7. Name the single-writer assumption. Name one incident that should move the dataset to a table format instead of a fancier marker.

Expected deliverables. The zone table, the tests, a sample marker, the partition statement, and the compaction rule.

Evaluation rubric.

You pass when a reviewer can point at one prefix a restatement may scan, and can point at the open hour it must not scan, without listing the bucket to decide.

Lesson Summary

A lake stores the bytes the mart may need again: landing locked down, raw at the contract grain on the prefixes Module 4 already writes, conformed and curated as published run prefixes. One object put is the atomic unit object storage gives you. A closed hour is a marker whose count matches an immutable run prefix, registered in the catalog only after that match, with retention that outlives the Kafka topic. Small files are compacted after the hour closes. The BI role reads curated. The marker is a reader rule with a single writer, not a snapshot and not a merge. The next lesson puts a real commit under those same zones.

Data Lakehouse Architecture is that commit: the same files, a metadata version readers agree on, and a clear line between the lake prefix and the table.

Further Reading

• Amazon S3 data consistency model — strong consistency for each object operation. The prefix is still a set of keys.
• Hadoop S3A committers — why Spark on object storage needs an explicit commit protocol, and the committers a current Hadoop release actually ships.
• Hive language manual: DDL — external tables and partitions. ADD PARTITION is the controlled registration path.
• AWS Glue Data Catalog — the catalog and the crawler. Inference is not a contract.
• Data Formats and Serialization — Parquet layout and the small-file warning this lesson turns into compaction.
• ETL and ELT Architecture — the zone names and the restatement rule the marker serves.
• Real-Time Data Quality and Monitoring — the handoff this layout is not allowed to blur with table freshness.
• Data Lakehouse Architecture — the commit that replaces the marker when its assumptions break.