Delta Lake Architecture and Operations

Data Engineering & Modern Data Platforms · Lesson 50 · Modern Data Platforms and Cloud

Lesson Overview

Data Lakehouse Architecture required a version a reader can pin, a commit that fails closed when the quality gate fails, and a publication row that does not chase "latest." Delta Lake is one format that stores that version. Its version is an integer. Its metadata is a directory named _delta_log next to the Parquet files. A job that lists the Parquet and ignores the log is back on the lesson 48 prefix, with extra JSON nearby that it did not read.

This lesson operates that log for TechLearningPro's purchase lines and for the completion table when Delta is the serving format. It covers the commit file, checkpoints, optimistic conflicts, schema enforcement, MERGE as the restatement, time travel, OPTIMIZE, and VACUUM. It uses the commands documented by the Delta Lake project. It does not invent table properties. Where Databricks adds a managed behavior, the text says so and leaves the platform walkthrough to Databricks Lakehouse Platform. Iceberg and Hudi are the comparison in the next lesson, not a second implementation here.

The purchase table is the running example because a retry that appends doubles revenue, and a vacuum that deletes files under a running reader looks like a corrupt hour. Completion uses the same protocol at a different grain. The raw Connect prefix stays files until a job commits closed hours into Delta. Connect is not assumed to write _delta_log.

Learning Objectives

By the end of this lesson you will be able to:

1. Read a Delta commit as an ordered log of actions under _delta_log, and explain what a checkpoint is for.
2. Describe optimistic concurrency well enough to predict when two writers conflict and when two appends can both commit.
3. Enforce a curated schema, and keep mergeSchema off the serving table.
4. Restate a day with MERGE on the business key so a retry does not duplicate rows.
5. Query a historical version, and set vacuum retention so that version still has its files.
6. Compact small files with OPTIMIZE without treating that rewrite as a new business publish.
7. Configure object-store commits so two clusters cannot both create the same version file.
8. Record txn identifiers for idempotent streaming batches, tying the log to the lesson 32 checkpoint discipline.

Prerequisites

• Lesson 11: Parquet files, row groups, and statistics. Delta stores those files. The log decides which of them are in the table.
• Lesson 13: replace-the-window restatement. MERGE is that rule on this format.
• Lesson 17: the gate that must pass before publication names a version.
• Lesson 32: Structured Streaming checkpoints and foreachBatch. Idempotent Delta writes use a transaction id beside that checkpoint.
• Lesson 47: dead-letter completeness. A Delta commit of a short hour is still short.
• Lesson 48: closed hours and markers. The Delta job reads those hours. It does not commit the open one.
• Lesson 49: publication names a snapshot. In Delta that snapshot is a version integer. Maintenance versions must not move publication by themselves.
• The Delta transaction log protocol, for anything this lesson summarizes. The summary is not a second spec.

Detailed Core Concepts

The table is a directory with a log

A Delta table is a directory of data files plus _delta_log. Data files are typically Parquet. The log is the source of truth for which of those files belong to the current version, what the schema is, and how partitions are declared.

Each commit is a JSON file whose name is a 20-digit, zero-padded version:

textCopy
_delta_log/00000000000000000000.json
_delta_log/00000000000000000001.json
_delta_log/00000000000000000010.checkpoint.parquet
_delta_log/_last_checkpoint

Version 0 creates the table. Version 1 is the next successful commit. Readers reconstruct state by starting at the newest checkpoint at or before the version they need, then applying the JSON commits after that checkpoint in order. Checkpoints exist so a reader does not replay the entire history on every query. delta.checkpointInterval defaults to 10 commits. _last_checkpoint records where the latest checkpoint is, as an optimization. A missing hint must not be required for correctness if the engine lists the log; do not hand-edit these files.

A commit contains actions. The ones you will meet in operations:

add does not mean "the bytes changed in place." Delta does not update a Parquet row. An update writes new files and removes the old ones in the same commit. A reader on the previous version still names the old files. That is copy-on-write at the file level. Deletion vectors, when enabled, are a later protocol feature that can record deleted rows without rewriting the whole file. They bump the protocol. Leave them off until every reader you run documents support. This lesson's operations assume classic add and remove, which every current Delta reader understands.

remove is logical. VACUUM is the physical delete, later, under a retention window. Mixing those two up is how time travel breaks.

What a reader actually opens

A query at the current version lists the log, loads the latest checkpoint, applies later commits, and builds the file list. It reads those Parquet files and no others, even if extra files sit in the directory. A query VERSION AS OF n stops at version n. A query TIMESTAMP AS OF resolves the commit whose timestamp is the latest at or before that time, using the timestamps in the log. Clock skew and commit-info timestamps are a reason to prefer version ids in the publication row from lesson 49. Timestamps are for humans investigating. Snapshot ids are for the API.

DESCRIBE HISTORY prints the commit sequence: version, timestamp, operation (WRITE, MERGE, OPTIMIZE, VACUUM, and others), and operation metrics when the writer supplied them. History is subject to delta.logRetentionDuration, which defaults to an interval of 30 days. After log cleanup, you cannot time-travel past what the log still describes, even if data files remain. Log retention and file retention are different properties. Set them together so the audit window is real.

Optimistic concurrency

A writer reads a version, writes new data files, and attempts to create the next JSON commit file. Creating that exact version name is the commit. If another writer created it first, this writer must not overwrite it. The loser recomputes against the new head or fails.

Whether the loser conflicts or can commit a further version depends on the isolation level and on what each transaction touched. Delta documents Serializable and WriteSerializable, set with the table property delta.isolationLevel. WriteSerializable is the default called out in the Databricks Delta documentation and is the level you should assume unless you set another. Under that default, two blind appends can both commit. A transaction that reads or rewrites files another transaction also rewrote conflicts. Partitioned appends that touch disjoint partitions are the usual way a high-volume ingest avoids fighting a restatement of an old day.

Do not memorize a blog's conflict matrix in place of the protocol. Do design for it:

• One restatement writer per event_date for completion.
• Ingest appends only into partitions the restatement is not rewriting, or ingest and restatement are the same job.
• On conflict, retry the window from the same source snapshot. Surface the conflict in the job log. A swallowed exception followed by a second append is a duplicate.

On HDFS, creating a version file exclusively is a native operation. On S3 it is not, unless you configure a log store that provides an atomic create. Older Delta deployments on S3 use S3DynamoDBLogStore so a DynamoDB table arbitrates the version. Newer Delta releases can use S3 conditional writes when the bucket and the runtime support them. The Delta storage documentation for your release is the setup you copy. The default single-driver log store is not a multi-cluster commit protocol. Two clusters writing one _delta_log without the documented store can both believe they committed, or one can overwrite a version. That is a torn table. Configure the log store before the second writer exists, not after the first corrupt version.

Schema enforcement and evolution

By default, a write whose schema does not match the table fails. An extra column, a missing column the table does not treat as nullable in the way the writer assumed, or a type change, stops the commit. That is the curated behavior lesson 49 required for attempt_number.

mergeSchema (and the equivalent automatic schema evolution option on a write) updates the table schema as part of the commit so the extra column lands. That is useful on a raw exploration table with an owner. It is a defect on learning.completion and on commerce.purchase_line, where the contract is reviewed code. A producer that adds a field belongs in the registry process from lesson 43, then a deliberate ALTER TABLE or a reviewed evolution, then a writer. A notebook with mergeSchema=true is not that process.

Type widening and column renaming are protocol-sensitive. Column mapping, which allows renames by id rather than by physical name, changes reader requirements. Do not enable it on a table a second engine reads until that engine's Delta support matrix says the protocol version is accepted. protocol in the log is the check. minReaderVersion and minWriterVersion are the table properties. Raising them is easy. Lowering them is not a weekend task.

Partition columns are part of metaData. Changing them is a new table, not an alter you discover in production. Choose event_date for purchase lines and for completion, matching the restatement window. Do not add learner_id as a partition. Lesson 48's cardinality argument still holds. Stats in add actions support data skipping on non-partition columns. delta.dataSkippingNumIndexedCols defaults to 32. Put filter columns early in the schema if you rely on skipping, and know that stats are collected on future writes, not retroactively, when you change the property.

MERGE is the restatement

MERGE matches a source to a target and inserts, updates, or deletes in one commit. For purchase lines the match key is (order_id, line_id). For completion it is (learner_id, lesson_id, attempt_number, definition_version). The source is the restatement query for one window, not the entire history, unless you are migrating.

A retry of the same MERGE with the same source is idempotent if the match covers the grain and the update sets the same values. A retry of INSERT or of mode("append") is not. Streaming writers that must append in micro-batches use the txn action instead: options txnAppId and txnVersion on the write. Delta records the application version in the log and ignores a retry that repeats the same version. That is the documented idempotent-write mechanism. The application version must advance only after a successful commit you will not repeat, and it must not advance past a batch you might retry. Align it with the offset or the window id you already store in the lesson 32 checkpoint. Two different app ids for one pipeline will both write.

COPY INTO is a Databricks SQL command for loading files. This lesson stays with Spark MERGE and appends, which are Delta's own interface. Lesson 52 can place COPY INTO in the workspace. Do not assume every open-source Delta runtime accepts Databricks SQL extensions.

OPTIMIZE and Z-order

OPTIMIZE rewrites many small files into fewer larger ones and commits add and remove actions. The data values do not change. DESCRIBE HISTORY shows the operation. Lesson 49's rule: do not move the publication row because an OPTIMIZE commit appeared. Readers of the published version keep their files until they refresh. New readers of the current table see the compacted files. After vacuum retention, the small files go away.

OPTIMIZE ... ZORDER BY (columns) is a documented layout rewrite that colocates related values so data skipping can drop more files. It is still a rewrite, it still costs a scan, and it is not a partition. Choose columns the queries filter on that are not already the partition, such as course_id on completion. Z-ordering every column is a tax. Liquid clustering is a Databricks table feature, not a requirement of the open protocol this lesson teaches. If you are on Databricks and considering it, read lesson 52 and the current clustering documentation, and do not mix folklore about Z-order with a clustering key you have not tested.

OPTIMIZE on a table the sink is still appending to will conflict or will immediately be followed by more small files. Compact closed partitions. The open hour is still the streaming path.

VACUUM and retention

VACUUM deletes files that are not part of the latest version and are older than the retention threshold. It does not delete _delta_log. The default threshold is 7 days, from delta.deletedFileRetentionDuration (interval 1 week). Log files are cleaned separately on checkpoint, under delta.logRetentionDuration (default interval 30 days).

Time travel to a version that still needs a removed file fails once vacuum has deleted that file. A reader that started under the old version and is still running can fail if you vacuum inside its lifetime. Delta's own retention text says the file retention should exceed the longest concurrent reader or writer, and should exceed any streaming query downtime you expect to recover from. The safety check that refuses very short retention exists so VACUUM ... RETAIN 0 HOURS is not a casual command. Disabling the check to make a storage graph drop during an incident deletes the audit. Set delta.deletedFileRetentionDuration to the lesson 49 audit window if that window is longer than a week, and set log retention at least as long, then vacuum on a schedule with the default threshold or with an explicit RETAIN that matches the property.

VACUUM does not rewrite files. It does not fix small-file query performance. OPTIMIZE does that. Vacuum after optimize, outside the retention window, reclaims the removed files. The order is optimize, wait out retention if you still need time travel to the pre-optimize version, then vacuum. If you need to reclaim faster, you are choosing a shorter audit. Write that choice down. Do not discover it from a missing version on Thursday.

Change data feed, briefly

delta.enableChangeDataFeed is a documented table property. When enabled, Delta can record row-level changes for later incremental reads. It is off by default. Enable it only if a downstream replica, such as the Snowflake copy from lesson 49, will consume it, and budget the extra files. A replica that runs MERGE from a full window does not need the feed. The feed is not a substitute for the publication row. It is a change log of the table, including versions you did not publish, unless the consumer filters on publication.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph source [Closed inputs]
        marker[Lesson 48 marker]
        dlq[Purchase dead letters]
        gate[Lesson 17 gate]
    end

    subgraph delta [Delta directory]
        parquet[Parquet data files]
        log[_delta_log commits]
        ckpt[Checkpoint parquet]
    end

    subgraph serve [Serving]
        ver[Version N]
        pub[Publication row names N]
        opt[OPTIMIZE creates N plus 1]
    end

    marker --> gate
    dlq --> gate
    gate --> log
    log --> parquet
    log --> ckpt
    log --> ver
    ver --> pub
    opt --> log

The gate sits in front of the log. OPTIMIZE writes the log and does not write the publication row.

mermaidCopy
sequenceDiagram
    participant W as Writer
    participant S3 as Data files
    participant L as _delta_log
    participant R as Reader on version N

    W->>S3: Write new Parquet files
    W->>L: Create version N+1 JSON exclusively
    alt Version file created
        L-->>W: Commit accepted
        Note over R: Still lists files from N
    else Version file already exists
        L-->>W: Conflict
        W->>W: Retry against new head
    end

The exclusive create is the protocol. On S3 that create has to be the log store your release documents, not a plain overwrite.

Technical Explanation

Reconstructing version 3

Assume commits 0 through 3 and a checkpoint at 0 only, for a small table. Version 0's metaData sets the schema and event_date as the partition column. Version 1's add actions name the first load. Version 2's remove plus add is a MERGE that replaced one day's files. Version 3 is OPTIMIZE on that day: remove the small files, add the compacted files, data values unchanged. A reader at version 1 never opens version 2's files. A reader at version 3 never opens the removed small files, but vacuum has not deleted them yet if retention has not elapsed. DESCRIBE HISTORY shows four rows. Publication still names version 2 if nobody has decided that optimize changes the official snapshot. For completion, publication can stay on 2, and analysts who query the table head see 3. That split is correct only if 3 is value-identical to 2. Confirm operation is OPTIMIZE before you tell an auditor the head is safe to read without a publication move. A MERGE that landed as version 3 is not optimize, and publication must move only through the gate.

Conflict you should be able to narrate

The morning completion job reads version 40 and rewrites event_date=2026-11-10. A backfill notebook reads version 40 and rewrites the same partition. One create of 00000000000000000041.json succeeds. The other writer's commit fails. If the notebook catches the error and appends instead, you now have version 42 with duplicate grain. The repair is to fail the task, not to catch and append. The retry reads version 41 and merges again. If both jobs compute the same rows from the same raw markers, the second merge changes nothing. If they computed different definitions, you have a product argument, not a Delta argument. Stop the notebook's credentials either way.

Commands you will actually run

History and time travel:

sqlCopy
DESCRIBE HISTORY commerce.purchase_line;

SELECT order_id, line_id, amount_cents
FROM commerce.purchase_line VERSION AS OF 41
WHERE event_date = DATE '2026-11-10';

VERSION AS OF takes the integer you stored on the publication row. Keep that integer. Do not recompute "what Monday saw" from a timestamp typed from memory if you have the id.

A merge for one day, in SQL available on Delta runtimes:

sqlCopy
MERGE INTO commerce.purchase_line AS t
USING staged_lines AS s
ON t.order_id = s.order_id
 AND t.line_id = s.line_id
WHEN MATCHED THEN UPDATE SET
  amount_cents = s.amount_cents,
  event_time = s.event_time,
  event_date = s.event_date,
  load_id = s.load_id
WHEN NOT MATCHED THEN INSERT (
  order_id, line_id, course_id, amount_cents, event_time, event_date, load_id
) VALUES (
  s.order_id, s.line_id, s.course_id, s.amount_cents, s.event_time, s.event_date, s.load_id
);

The UPDATE list is explicit so a new source column does not appear in the serving table by accident. UPDATE SET * is shorter and will copy every matched column, including one you meant to quarantine. Prefer the explicit list on curated tables.

Maintenance:

sqlCopy
OPTIMIZE commerce.purchase_line
WHERE event_date = DATE '2026-11-10';

VACUUM commerce.purchase_line RETAIN 168 HOURS;

168 hours is the 7-day default, written out so the job review sees the number. If the audit window is 30 days, set the table property and retain 720 hours. Do not vacuum with a shorter retain than the property you told the metric owner about.

Retention properties:

sqlCopy
ALTER TABLE commerce.purchase_line SET TBLPROPERTIES (
  'delta.deletedFileRetentionDuration' = '30 days',
  'delta.logRetentionDuration' = '30 days',
  'delta.isolationLevel' = 'WriteSerializable'
);

Confirm property names against the Delta table-properties reference for your release before applying them in production. The names above are the documented ones as of the reference this lesson was written against. A typo in a property name can be ignored by the writer and leave you on the default.

Idempotent micro-batch append

For a streaming load of closed files into an append-only raw Delta table, where the batch id is the hour's run_id hashed to a monotonically increasing integer you control:

pythonCopy
# delta-spark APIs: format("delta"), option txnAppId, option txnVersion.
# Match the delta-spark build to Spark using the Delta release matrix.
(
    batch_df.write.format("delta")
    .mode("append")
    .option("txnAppId", "tlp-progress-raw")
    .option("txnVersion", batch_version)
    .save(progress_raw_path)
)

A retry of the same batch_version does not duplicate. The next hour uses a greater batch_version. Reusing a version for a different hour's data is a silent skip. The id is a contract, stored next to the marker run_id.

Schema enforcement on that write is the default. Leave mergeSchema unset. A new field fails the batch and pages the contract owner.

Statistics and the files OPTIMIZE will not save you from

Data skipping uses min and max stats stored with add. A file whose course_id range is the entire catalog will not be skipped by a course filter. That happens when each file is a random sample of courses, which is what a sink without a sort produces. ZORDER BY (course_id) on a closed partition rewrites files so a single-course query opens fewer of them. Measure bytes read before and after on a real query. If the filter is always event_date, the partition is enough and Z-order is optional. If the studio API reads one course, layout matters. The API should still go through the published version, not through a path someone z-ordered in a scratch table.

Practical Implementation or Design Walkthrough

Problem

Commit closed purchase-line hours into a Delta table keyed by (order_id, line_id), restate a day idempotently, publish the version, and compact without moving publication.

Setup assumptions

Spark with a matching delta-spark package, a warehouse directory on object storage or on a lab filesystem, and a single writer in the lab. On S3, the log store from the Delta S3 documentation is configured before two writers exist. The source is a DataFrame of one closed hour that already passed object-count and dead-letter completeness.

Steps

1. Create the table with the contract columns and PARTITIONED BY (event_date). Include load_id. Do not include payment fields lesson 42 kept out of the topic.
2. Append or merge the closed hour. Record the version from history where operation is the write you just ran.
3. Run the lesson 17 checks as queries on that version, not on the head after a later commit. Uniqueness of (order_id, line_id), not-null keys, and the completeness total you computed before the write.
4. If the checks pass, insert the publication row with that version. If they fail, do not publish. A following commit may repair. The bad version remains in history inside retention, which is what you want for the incident, and it is not official.
5. Retry the merge with the same source. Row counts per key stay at one. History gains a version. Publication stays put if the values did not change and the previous publication already named a good version. If your policy publishes every successful merge, the new version is also value-identical and either id is servable. Pick a policy and test it. The dangerous policy is publishing whatever max(version) is after a concurrent optimize or a failed experiment.
6. OPTIMIZE the partition. Confirm history. Confirm a VERSION AS OF the published id still returns the pre-optimize rows if you have not vacuumed. Confirm counts match the head if the only later operation is OPTIMIZE.
7. Attempt VACUUM with the retention you promised. Confirm a version inside the window still reads. Do not run a zero-hour vacuum to see what happens on a table you care about. A scratch table is the place to learn that the files disappear.
8. Write the conflict drill: two jobs, same partition, one must fail and retry. Capture the exception class your runtime raises so the runbook names it.

Expected lab evidence

History shows create, merge, a retry, and optimize as distinct operations. A duplicate-key query on the published version returns zero rows. A time-travel query on the published version succeeds before vacuum retention elapses. The publication table does not update when optimize runs, under the policy you wrote in step 5. The open hour is absent from the table.

Enterprise Real-World Scenarios

The retry that appended

The purchase merge job fails after Delta has committed and before the task reports success to the orchestrator. The orchestrator retries. An older version of the job catches the conflict and, on any exception, writes mode("append") "so the hour is not lost." The hour is now in the table twice. Revenue for the promotion is high. The uniqueness gate, if it runs after publication, is too late. If it runs before publication and someone published on the first success already, the API is serving duplicates until a repair merge deletes them.

The repair merge deletes the duplicate grain, keeping one load_id, in a new version, after a human checks which load_id matches the source marker. Publication moves only after the uniqueness query returns zero. The code change removes the append fallback. Retries call the same MERGE. Streaming paths that truly append use txnAppId and txnVersion tied to the hour's run id. The incident note says the commit can succeed when the orchestrator thinks it failed. Idempotency has to live in the log, not in the hope that the task failed before the JSON file existed. That is the same class of bug as a database commit the client never saw.

Vacuum during the audit question

An auditor asks what completion the API served last Tuesday. The publication row has version 88. VERSION AS OF 88 fails because the files were removed. A storage cleanup job ran VACUUM ... RETAIN 1 HOURS with the retention check disabled, during a cost push, two days after optimize had removed the old files logically. Log retention still lists version 88. The data files do not exist. Time travel's metadata and time travel's bytes have come apart.

The repair is procedural because the bytes are gone: rebuild Tuesday from raw markers and the definition version, and label the rebuild as a reconstruction, not as version 88. Then set file retention and log retention to the audit window, restore the vacuum safety check, and deny the cleanup role the ability to set a shorter retain. OPTIMIZE stays on the schedule. Vacuum stays on the schedule at 30 days, or whatever the metric owner signed. Cost reviews can see the retained bytes as the price of the audit, not as waste. Lesson 49 already said expiry is not an incident response. This is the incident that sentence was about.

Best Practices

• Treat _delta_log as the table. Forbid jobs that glob the Parquet directory and skip the log.
• Configure the S3 log store from the Delta documentation for your release before a second cluster writes.
• Keep one restatement writer per partition window. Retry MERGE. Do not fall back to append.
• Use txnAppId and txnVersion for append pipelines whose orchestrator can retry after a commit.
• Leave schema evolution off on curated tables. Change schema in a reviewed alter.
• Partition by the restatement date. Put high-cardinality ids in columns, and rely on stats or a measured Z-order.
• Store the Delta version on the publication row. Do not publish max(version).
• Let OPTIMIZE run on closed partitions. Do not move publication for a values-preserving optimize.
• Set log retention and deleted-file retention to the same audit window. Vacuum on a schedule at that window.
• Keep deletion vectors and column mapping off until every reader documents the protocol.
• Run quality checks against the version you will publish, then publish that integer.
• Put dead-letter completeness in front of the purchase merge, as in lesson 47.
• Read DESCRIBE HISTORY in the incident before you rewrite. The operation column distinguishes optimize, merge, and vacuum.

Common Mistakes and Anti-Patterns

Reading the directory. Extra Parquet from a failed job, or files remove has logically dropped, re-enter the count. Query the table or an explicit version.

Two clusters and the default S3 log store. Version files get overwritten or duplicated in spirit. Set the documented multi-writer store first.

mergeSchema on the serving table. A bad job adds a column and the gate's explicit column list no longer matches what analysts see. Fail the write. Evolve on purpose.

Append as the retry path. The first commit may have landed. The retry doubles the grain. Merge, or use txn versions.

Vacuum shorter than the audit, especially with the safety check off. History still names versions whose files are gone. That is a false audit trail.

Optimizing the open partition. You fight the ingest and you compact files the next micro-batch will not reuse. Compact closed dates.

Publishing the head after maintenance. The API starts serving an experiment that committed a minute ago. Publication is an explicit row.

Enabling a new protocol feature for a performance blog. Older Spark jobs fail on protocol. Test every reader, including the replica job, before you raise minReaderVersion.

UPDATE SET * from a staging frame that still has a prohibited column. The serving table gains the column the boundary job existed to drop. Name the columns.

Z-order on a schedule that rewrites the whole table nightly. You pay a full rewrite so a filter you do not run can skip files. Partition prune first. Z-order a measured query.

Hand-editing JSON in _delta_log. The next checkpoint and the next writer will not respect the story you invented. Restore by committing a new version from raw, or by the documented restore operation if your runtime supports RESTORE, which Delta documents as RESTORE TABLE ... TO VERSION AS OF. Use that command. Do not splice the log.

Troubleshooting and Design Review

Symptom: VERSION AS OF the published id throws a file-not-found error.

Likely cause: vacuum deleted files that version still referenced, or a lifecycle rule on the bucket deleted Parquet under the table. Lifecycle rules that expire objects by age do not understand Delta remove actions. They are a second, unauthorized vacuum.

Investigation: history around that version, the table's deletedFileRetentionDuration, bucket lifecycle on the table prefix, and whether a job disabled the retention check.

Corrective action: stop lifecycle expiry inside the Delta prefix. Rebuild the served window from raw if the files are gone. Set retention to the audit window. A Delta table's files are garbage-collected by VACUUM only.

Symptom: Row counts doubled after a successful orchestrator retry.

Likely cause: append on retry, or a txnVersion that changed even though the batch was repeated, or two app ids.

Investigation: history for two writes of the same hour. The write option code. The txn actions if you can see them in the commit JSON.

Corrective action: merge to one row per grain. Fix the retry. Add the uniqueness query as a gate before publication so the next double never becomes official.

Symptom: A second job fails every morning with a commit conflict on event_date.

Likely cause: the backfill or the optimize job overlaps the restatement on the same files. This is the protocol working.

Investigation: history timestamps, which job holds the cluster, whether optimize is targeted at the partition being merged.

Corrective action: serialize them in the orchestrator, or target optimize at dates outside the restatement window. Do not lower isolation to make the error vanish. You would be choosing a weaker guarantee to hide a schedule clash.

Symptom: A Spark job that used to read the table fails after a feature flag, with a protocol version error.

Likely cause: column mapping, deletion vectors, or another writer feature raised minReaderVersion.

Investigation: the protocol action and the job's Delta library version. The change that enabled the feature.

Corrective action: upgrade the reader to a release that documents support, or rewrite the table to a protocol the reader supports if the feature was optional. Do not keep writing with the feature while a required reader is old. Decide the reader matrix before the next flag.

Symptom: Queries by course_id scan almost the whole day despite a partition filter on event_date being absent and the day filter being present...

Read it as: the query filters course_id and event_date, the date prune works, and the course filter still opens every file in the day.

Likely cause: no useful stats, or each file contains every course.

Investigation: the scan metrics in the Spark UI, file min/max for course_id if you sample add stats, whether Z-order or a sort has ever run.

Corrective action: if the query shape is real and frequent, OPTIMIZE ... ZORDER BY (course_id) on closed partitions and compare bytes scanned. If the query is rare, do not rewrite the table for it. An API that needs one course should hit a published aggregate at course grain if the scan is too expensive. That aggregate is a modeled table, not a layout trick.

A design review asks: where is _delta_log, and who can write it? Which log store runs on S3? What is the match key of MERGE? What happens on retry? Which version does publication store? What is the vacuum retention in days, and is bucket lifecycle excluded? Which protocol features are forbidden until the reader matrix is updated? Does optimize move publication?

Interview Questions and Answers

Q1. What is in _delta_log, and why is a checkpoint not the table?

JSON commits named by a zero-padded version, plus periodic checkpoint files, plus a hint at the last checkpoint. Each commit is a set of actions: metadata, protocol, add, remove, optional txn, commit info. A checkpoint is a rollup so readers do not replay from version 0. The table a reader sees is the checkpoint plus the commits after it, or the commits alone. If I delete the log and keep the Parquet, I have files, not a Delta table.

Q2. Why does an update rewrite files instead of editing a row?

Parquet files in a Delta table are immutable. An update writes new files and a commit that adds them and removes the old paths. Readers on the previous version still name the old paths, which is how time travel and concurrent readers work. Vacuum may delete the old paths only after retention. I do not open a Parquet file and patch a row in place.

Q3. Two appends succeed and two merges of the same partition do not. Is the log broken?

No. Under the default WriteSerializable behavior Delta documents, blind appends are allowed to commit together. Transactions that rewrite the same files conflict. The loser retries. I want that conflict on a completion restatement. I would be worried if both merges silently succeeded and interleaved one partition's files. I design one writer per window so the conflict is an exception path, and I still handle it.

Q4. How do you stop a retry from doubling purchase lines?

I MERGE on (order_id, line_id) so a second application sets the same values. If the pipeline must append, I set txnAppId and txnVersion to the hour's batch id, which Delta records in a txn action and ignores on repeat. I do not catch a post-commit failure and append. I run a uniqueness check on the version I intend to publish.

Q5. What is the difference between remove and VACUUM?

remove takes a file out of a version. The bytes remain so older versions and slow readers can still open them. VACUUM deletes bytes that the latest version does not need and that are older than delta.deletedFileRetentionDuration, default one week. Log retention is separate, default 30 days, and it limits how far DESCRIBE HISTORY and time travel can see. I set both to the audit window. I do not point an S3 lifecycle expiry at the table prefix.

Q6. Why must the publication row store a version integer rather than "current"?

OPTIMIZE, failed experiments, and repairs all commit new versions. Current is whichever finished last. The API needs the version the gate approved. I write that integer on the publication row from lesson 49. Time travel for an auditor uses the same integer. A timestamp is a fallback for humans, and I do not want the API to depend on clock interpretation.

Q7. What do you configure before two Spark clusters write one Delta table on S3?

A log store the Delta release documents for multi-cluster S3 commits: historically a DynamoDB-backed store, and on releases that support it, S3 conditional writes. I copy the configuration from that release's storage page, including the table name it requires. I do not run two clusters on the default single-driver store. Exclusive create of the version file is the whole protocol. S3's ordinary put is not that create.

Q8. When do you turn on mergeSchema?

On a scratch or raw exploration table whose schema is supposed to follow the files, with an owner. Not on learning.completion or commerce.purchase_line. Those schemas change through a reviewed alter after the registry or the mart contract changes. A failed write is the signal that the job and the table disagree. Auto-merge hides the signal inside a successful commit.

Q9. What does OPTIMIZE change, and what must it not change?

It changes file layout: fewer files, new paths, add and remove in a new version. It does not change row values. It must not update the publication row. I verify the history operation before I tell anyone the table head is business-equivalent to the published version. If any later operation is a MERGE or DELETE, head and published are different products until publication moves.

Q10. A streaming query died for a day and the checkpoint points at a Delta version whose files were vacuumed. What retention did you violate?

delta.deletedFileRetentionDuration has to cover the longest downtime from which that stream must recover, because the stream may still need the files of the snapshot it started from. Delta's retention notes say this directly. I size retention for the audit and for that recovery, not for the happy path where the stream stays up. Then I vacuum inside that size.

Q11. How does Delta's schema sit next to the Kafka registry?

The registry gates producers of learning.progress and commerce.purchase_line. The Delta schema gates the curated or conformed table. They should match on the columns the table claims to store, and the table is allowed to drop fields the topic still carries if the contract says so. I do not let a Delta evolution add a column the registry rejected, and I do not let a registry change silently widen the table. Two reviews, one column list.

Hands-On Exercises

Exercise: One published version, one retry, one vacuum policy

Problem statement. Build a lab Delta table for purchase lines. Merge a closed hour, retry it, publish a version integer, optimize, and show that vacuum retention is an audit decision.

Tasks.

1. Create a partitioned Delta table with an explicit column list. Document the delta-spark and Spark versions you paired, using the release matrix.
2. MERGE a batch of ten lines. Run the merge again. Show one row per (order_id, line_id) and two history entries.
3. Implement a publication record, in a table or in a test double, that stores the version. Show that a later OPTIMIZE does not change the stored version under your policy.
4. Query VERSION AS OF the published version and the head. Counts match if only optimize followed.
5. Add a second writer that merges the same partition concurrently. Capture the conflict and show the retry path is merge, not append.
6. Write the txnAppId approach as an alternative for an append-only raw table, and state how batch_version must move.
7. Set a retention property for a 30-day audit. State why a bucket lifecycle rule must not expire the prefix. Do not disable the vacuum safety check.
8. State how dead-letter completeness gates the merge.

Expected deliverables. History output, the uniqueness query result, the publication note, the conflict log, and the retention decision.

Evaluation rubric.

You pass when a reviewer can point at the version the API would serve, the history row that must not change that pointer, and the retention that keeps the version readable.

Lesson Summary

Delta Lake stores a table as Parquet files plus _delta_log. Versions are integers. Commits add and remove files. Readers pin a version. Writers create the next log file exclusively, and on S3 that exclusivity is a configured log store. MERGE on the grain makes restatement retries safe. txn versions make append retries safe. Schema enforcement stays on for serving tables. OPTIMIZE rewrites layout and must not move publication. VACUUM deletes old files after delta.deletedFileRetentionDuration. Log retention is a second clock. Both match the audit window. Bucket lifecycle is not a vacuum you control. Publication stores the version the gate approved.

Apache Iceberg and Hudi Fundamentals is the same promise with a different metadata tree, a different writer model, and a real choice when more than one engine must commit or when the workload is a keyed upsert stream. The publication rule does not change.

Further Reading

• Delta Lake protocol — actions, versions, and conflict behavior. This lesson is an operator's reading, not a replacement.
• Delta Lake table properties — logRetentionDuration, deletedFileRetentionDuration, checkpointInterval, isolationLevel, enableChangeDataFeed.
• Table utility commands — history, vacuum, restore, and the warning that vacuum ends time travel past the retention window.
• Delta Lake on object storage — log stores for S3 and other systems. Copy the configuration for your release.
• Idempotent writes — txnAppId and txnVersion, if the page anchor moves, search the streaming section of the same manual.
• Data Lakehouse Architecture — publication versus current version.
• Databricks Lakehouse Platform — workspaces, SQL warehouses, and managed maintenance around this format.
• Apache Iceberg and Hudi Fundamentals — when the log you just learned is not the format you should operate.