Apache Iceberg and Hudi Fundamentals

Data Engineering & Modern Data Platforms · Lesson 51 · Modern Data Platforms and Cloud

Lesson Overview

Delta Lake Architecture and Operations made a version integer and a _delta_log directory concrete. TechLearningPro still has to choose a format when the serving engines are not a single Databricks runtime. Finance wants the completion table in a SQL engine that already speaks Iceberg. The enrollment current-state job wants to upsert by primary key every few minutes and then read only the keys that changed. Those are not the same workload, and they are not a reason to run three formats on one grain.

Apache Iceberg and Apache Hudi both store table state as metadata beside data files in object storage. Iceberg's metadata is a tree: a metadata file, a snapshot, a manifest list, manifests, and data files, with column ids and hidden partitions. Hudi's metadata is a timeline of instants under .hoodie, organized as file groups, with a record key and a precombine field that decides which write wins. This lesson teaches those mechanisms well enough to operate them and to reject a bad catalog, a bad precombine, or a delete-file pile. It uses the Iceberg spec and the Hudi overview's documented concepts. It does not invent Spark procedure arguments. Named arguments below are the ones the Iceberg Spark procedure docs publish; confirm them when you upgrade.

Delta remains the right operational choice when lesson 52's platform is the only writer and the only reader that matters. This lesson does not re-teach the Delta log. It gives you the comparison the platform choice needs.

Learning Objectives

By the end of this lesson you will be able to:

1. Walk an Iceberg snapshot from the catalog pointer through the manifest list to data files, and say what a reader pins.
2. Explain column ids and hidden partition transforms, and why renaming a column is safe when the id stays.
3. Distinguish copy-on-write rewrites from position and equality delete files, and schedule compaction and snapshot expiry.
4. Refuse a Hadoop catalog on S3 for a production table, and name a catalog that commits the metadata pointer.
5. Explain a Hudi timeline, file group, and the difference between copy-on-write and merge-on-read.
6. Choose a record key and a precombine field that will not drop a correction.
7. Compare Delta, Iceberg, and Hudi for one TechLearningPro grain using writer shape, engines, and maintenance.
8. Keep lesson 49's publication row, regardless of which metadata tree you adopt.

Prerequisites

• Lesson 11: Parquet, statistics, and schema evolution at the file level. Iceberg evolution is by column id, which files alone do not give you.
• Lesson 16 and lesson 46: change data and primary keys. Hudi's record key is that key, with a writer model aimed at upserts.
• Lesson 25: object storage does not rename a directory atomically. The catalog commit is the replacement for that rename.
• Lesson 48: partitions you would otherwise encode in the path. Iceberg can hide that encoding.
• Lesson 49: one server, one publication snapshot, maintenance that must not flip Monday.
• Lesson 50: optimistic commit, vacuum versus logical remove, and MERGE idempotency. The same operational morals apply with different file names.
• The Iceberg spec and the Hudi docs overview, when a detail here is only a map of those documents.

Detailed Core Concepts

Iceberg's tree is the snapshot

An Iceberg table's current state is a pointer, held by a catalog, to one metadata JSON file. That file records the schema list, the partition specs, the current snapshot id, and the snapshot log. A snapshot points at one manifest list. The manifest list points at manifest files. Each manifest points at data files and, in format v2, at delete files, with partition data and column lower and upper bounds.

A reader that is given snapshot id S reads that tree and nothing else. A writer creates new data files and a new metadata file whose current snapshot is S', then asks the catalog to swap the pointer from the old metadata file to the new one if the pointer is still the one the writer started from. That swap is the optimistic commit. If the catalog has already moved, the writer retries. Two engines agree on the table when they use the same catalog. They do not agree because they share a bucket.

The Hadoop catalog stores that pointer as files in the warehouse directory, including a version-hint style pointer. Iceberg's own documentation treats the Hadoop catalog as a simple implementation, not a production catalog. On S3, where a pointer file is an object overwrite, two writers can lose updates. Do not use it for learning.completion or for purchase lines. Use a catalog with a conditional pointer update: the Hive metastore, the AWS Glue catalog, a JDBC catalog, or an Iceberg REST catalog. Nessie and other REST implementations are catalogs. They are not separate table formats. The spec's REST catalog is the interface to ask for when a second engine, Spark on one side and Trino or Snowflake on the other, must see one pointer.

Column ids, not column positions

Iceberg assigns a field id when a column is added. Parquet files store the column; the metadata maps field ids to names. Renaming learner_id in the table schema does not rewrite files. Dropping a column hides it from new readers and leaves old files intact. Adding a column assigns a new id. Readers of old files see null or the default the spec requires for that change. Reordering columns is metadata. Promoting an int to a long, or a float to a double, is the kind of widening the spec allows. A string where there was a long is not an evolution. It is a new column or a new table.

This is stricter and kinder than a Hive table, which often treats the Parquet name or the position as the schema. Lesson 11's warning about a renamed field becoming a silent null is the Hive-shaped failure. Iceberg avoids it when every writer goes through the table API. A job that drops raw Parquet into the table directory and hopes the names line up bypasses ids. Load through the writer.

Partition evolution uses the same idea. A partition spec has an id. Old files keep the spec they were written with. New files use the new spec. A reader plans both. You can add a bucket transform later without rewriting history, at the cost of a more interesting plan. "Interesting" means you test a query that spans the change. It does not mean the feature is free.

Hidden partitioning

Hive-style paths from lesson 48 put event_date=2026-11-10 in the key because the engine only prunes what it can see in the directory name. Iceberg stores partition values in the metadata. The transform is part of the spec: identity, year, month, day, hour, bucket[N], truncate[W]. A table partitioned by days(event_time) accepts WHERE event_time >= ... and prunes without a separate event_date column the writer must populate. The column you filter is the column you stored.

That removes a class of bugs where event_date and event_time disagree. It does not remove the need to choose a transform that matches the query. days(event_time) matches a daily restatement. hours(event_time) matches the closed-hour ingest and creates more partitions. bucket(256, learner_id) spreads writes and does not answer "all events on Tuesday" cheaply. Do not bucket by learner because a tutorial did. Partition for the filter the publish job and the API actually use.

Partition columns do not have to appear as redundant data columns. They also must not be the only place a value lives if a maintenance rewrite changes the spec and you still need the timestamp. Keep event_time. Derive the partition from it.

Copy-on-write and delete files

Format v1 tables rewrite data files to apply a delete or an update. Format v2 adds delete files. A position delete names a file and a row position. A equality delete names column values, such as an event_id, that must be removed wherever they appear in the snapshot's data files, subject to the sequence rules the spec defines. Merge-on-read means readers apply those delete files at read time. Copy-on-write means the writer rewrote the data files so the new snapshot has no delete to apply for that change.

Equality deletes are attractive for an upsert that does not know the Parquet row position. They are also how a read gets slower every hour: each snapshot carries more delete files, and every scan applies them. Compaction rewrites data files and drops delete files that are no longer needed. In Spark, rewrite_data_files is the documented procedure. Position-delete compaction has its own procedure, rewrite_position_delete_files, when that is the pile you have. Read the procedure page for the release you run before you schedule five of them you do not need.

Sequence numbers on snapshots decide which delete applies over which data. You do not hand-assign them. You do avoid a second writer that bypasses the API and drops files into the directory. The sequence is the writer's job.

Expiry is Iceberg's vacuum

expire_snapshots removes snapshots older than a timestamp, while retaining a minimum number of ancestor snapshots (retain_last, defaulting in the procedure docs). Files that belong only to expired snapshots can be deleted. Files a retained snapshot still names cannot. The main branch does not expire. Branches and tags can pin snapshots for audits; by default their retention is long, and the table property history.expire.max-ref-age-ms changes tag expiry. A tag named monday-2026-11-10 is a better audit handle than a timestamp remembered in chat, and it will also stop expiry from deleting that snapshot until you drop the tag or its age policy elapses. Use a tag for the published snapshot if you want the catalog to enforce what lesson 49 asked the publication row to remember. The publication row remains the API's switch. The tag is the expiry pin. They should name the same snapshot. If they drift, the API wins and the tag is a bug.

Compaction writes new files and a new snapshot first. Expiry later removes the uncompacted snapshot. Expiring before anyone has rewritten, with retain_last of 1 and a short older_than, is the same incident as Delta VACUUM inside the audit window. Set older_than to the audit window. Keep retain_last large enough for the morning's retries.

Hudi's timeline and file groups

A Hudi table is a directory with data files and a .hoodie folder. The timeline is an ordered log of instants. Each instant has an action, such as commit, deltacommit, compaction, clean, or savepoint, and a state: requested, inflight, or completed. A reader of a completed commit sees that instant's result. An inflight instant is not a version you publish. Cleaning removes old file versions beyond a retention policy, analogous to vacuum. A savepoint pins an instant so cleaning does not remove it. Savepoints are the Hudi-shaped audit pin. Use one for the published instant if this format serves completion. Still record the instant time on the publication row.

Data is organized into file groups. A file group has a base file, usually Parquet, and may have log files. The record key identifies the row inside a partition. The partition path is the Hive-style path Hudi writes unless you configure otherwise. The precombine field, also called the ordering field in newer docs, decides which of two records with the same key is newer.

Copy-on-write rewrites the base file when a key in that file group changes. Reads are simple Parquet scans of base files. Writes cost more when a single key update rewrites a large file. Merge-on-read appends updates to log files beside the base and later compacts logs into a new base. Delta commits (deltacommit) complete quickly. Readers either see compacted data only, or they merge logs on read, depending on the query type you set. Compaction is a separate instant you must schedule. A merge-on-read table without compaction becomes a pile of logs, and every read pays it.

Hudi's index finds which file group holds a key so an upsert does not scan the table. The Bloom index is the long-standing default on many Spark paths. A bucket index is a deterministic placement you can plan for. The index is part of correctness. If it misses a key, the upsert inserts a second copy. Operate the index the Hudi write-path page for your version describes. Do not turn indexing off to make a benchmark faster and then call the result an upsert table.

Precombine is a business rule

When two records share a record key, Hudi keeps the one with the greater precombine value. If you set precombine to event_time and a correction arrives with an earlier event_time and a later processing sequence, the correction loses and the wrong state remains. Enrollment current state is the sharp case. A row's key is the enrollment primary key. The ordering field must be a monotonic update sequence from the source, such as the log sequence or the Debezium source timestamp only if a correction cannot carry an older timestamp. Lesson 46's source.ts_ms is a commit time on the database change, which usually moves forward. A snapshot reload that replays old source.ts_ms values can still go backwards relative to a newer change. Prefer an explicit monotonically increasing version from the CDC event if you have one, and test a replay.

Progress events are a poor Hudi table if the grain is append-only event_id. You do not need upserts. An Iceberg or Delta append, or the raw prefix, is the simpler machine. Hudi earns its place on keyed, mutable tables: enrollment current state, a slowly changing dimension you refuse to model as lesson 10's history and instead keep as "current," or a purchase line you truly update in place. Purchase lines in this course are insert-only at the source. Merging them in Delta or Iceberg on (order_id, line_id) is enough. Hudi is justified when the update rate and the incremental consumer are the workload, not when the team wants a third logo.

Incremental reads

Hudi documents an incremental query: a start instant and an end instant, returning records written between them. That is the replica feed, if Hudi is the server. Consumers must store the last instant they applied, the way a Kafka consumer stores an offset. Reprocessing the same instant range must be idempotent on the destination, because at-least-once delivery of a batch is the normal retry. Iceberg incremental reads exist as well, from one snapshot to another, in engines that implement them. Delta's change data feed was the lesson 50 version of the same idea. Pick the incremental mechanism of the format you picked. Do not bolt a second format onto the table to get incremental reads.

How to choose, for this platform

One grain, one row of that table. Completion that is already a daily MERGE can live in Iceberg if the SQL engine is not Databricks, or in Delta if it is. It should not live in Hudi unless you have measured an upsert and incremental problem the merge does not have. Enrollment current state can live in Hudi if that job is the server, or as an Iceberg MERGE if you would rather staff one format. Two formats means two on-calls. Lesson 49 already forbade three formats for one grain. This table is how you pick the one.

Snowflake native tables, lesson 53, remain an alternative server that is not an open format at all. Iceberg in Snowflake is a documented interoperability feature with limits that change by release. If Snowflake is the server, prefer its native table and treat Iceberg as a sharing mechanism you have tested, not as an assumption that every Snowflake feature exists on the external table.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph catalog [Catalog commit]
        cat[REST Glue Hive or JDBC]
        meta[Iceberg metadata JSON]
    end

    subgraph tree [Snapshot tree]
        snap[Snapshot id]
        mlist[Manifest list]
        mans[Manifests]
        data[Data files]
        dels[Delete files]
    end

    subgraph hudi [Hudi timeline]
        tl[.hoodie timeline]
        base[Base Parquet]
        logs[MOR log files]
        idx[Index]
    end

    cat --> meta
    meta --> snap
    snap --> mlist
    mlist --> mans
    mans --> data
    mans --> dels
    tl --> base
    tl --> logs
    idx --> base

Iceberg readers start at the catalog. Hudi readers start at the latest completed instant. Neither starts at a bare prefix list.

mermaidCopy
sequenceDiagram
    participant W as Iceberg writer
    participant F as Object storage
    participant C as Catalog
    participant R as Reader on snapshot S

    W->>F: Write data files and a new metadata file
    W->>C: Swap pointer if current is still S
    alt Swap accepted
        C-->>W: Current snapshot is S-prime
        Note over R: Still reads S
    else Pointer moved
        C-->>W: Conflict, retry
    end

The catalog swap is the whole multi-engine agreement. A version-hint overwrite on S3 is not this sequence.

Technical Explanation

A Spark catalog that can be shared

textCopy
spark.sql.catalog.tlp = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.tlp.type = rest
spark.sql.catalog.tlp.uri = https://iceberg-catalog.internal.example
spark.sql.catalog.tlp.warehouse = s3://tlp-lakehouse/iceberg

type=rest is the documented catalog type for the REST protocol. The URI is your catalog, not a public endpoint this lesson invents. type=glue is the documented AWS Glue catalog type when Glue holds the pointer. type=hive talks to a Hive metastore. type=hadoop is the one you use in a single-process lab and do not point at a shared S3 bucket for completion. Warehouse IAM belongs to the job role. Human roles query through an engine that uses the catalog, as in lesson 49.

Create the completion table with a hidden day transform:

sqlCopy
CREATE TABLE tlp.learning.completion (
  learner_id string,
  lesson_id string,
  attempt_number int,
  definition_version string,
  completed_at timestamp,
  event_time timestamp
)
USING iceberg
PARTITIONED BY (days(event_time));

event_time on completion is the business time the restatement already uses, so a restatement of Tuesday prunes with a filter on that column. Confirm the Spark SQL DDL against the Iceberg Spark docs for your version if USING iceberg is expressed differently in your runtime. The partition transform name days is the spec's.

Publish by snapshot id:

sqlCopy
SELECT snapshot_id
FROM tlp.learning.completion.snapshots
ORDER BY committed_at DESC;

The snapshots metadata table is a documented Iceberg feature. Store the id you just committed on the publication row after the lesson 17 gate. Readers that honor time travel use VERSION AS OF <snapshot_id> or a tag. Confirm the time-travel clause in the engine you serve with. Spark and Trino both document snapshot and timestamp travel. They do not document "whatever is in the directory."

Maintenance procedures

From the Iceberg Spark procedure reference, named arguments, which that page recommends:

sqlCopy
CALL tlp.system.rewrite_data_files(
  table => 'learning.completion',
  strategy => 'binpack',
  where => 'event_time >= TIMESTAMP ''2026-11-01 00:00:00'''
    AND event_time < TIMESTAMP ''2026-11-11 00:00:00'''
);

CALL tlp.system.expire_snapshots(
  table => 'learning.completion',
  older_than => TIMESTAMP '2026-10-12 00:00:00',
  retain_last => 10
);

strategy => 'binpack' is the default compaction strategy in that reference. sort and Z-order or Hilbert options exist on rewrite_data_files for engines and releases that list them. Use them when a measured filter needs layout, the same judgment as Delta Z-order. Do not expire snapshots a publication tag or row still needs. retain_last => 10 keeps ten ancestors even if they are older than older_than. It is not a substitute for a 30-day audit if those ten snapshots cover only an afternoon. Pick older_than from the audit window first.

rewrite_manifests rewrites manifests when planning gets heavy because manifests are many and small. Schedule it when planning time shows up in the query profile, not on a superstition.

A Hudi upsert that states its ordering

The Spark data source options below are the ones Hudi's quickstart has documented across the 0.x line. Hudi 1.x still describes the same writer concepts. Check the config page if a key alias appears.

pythonCopy
# Record key is the enrollment primary key.
# precombine must be monotonic for corrections, not a business timestamp
# that a backfill can move backwards.
hudi_options = {
    "hoodie.table.name": "enrollment_current",
    "hoodie.datasource.write.recordkey.field": "enrollment_id",
    "hoodie.datasource.write.partitionpath.field": "course_id",
    "hoodie.datasource.write.precombine.field": "change_sequence",
    "hoodie.datasource.write.table.type": "COPY_ON_WRITE",
    "hoodie.datasource.write.operation": "upsert",
    "hoodie.datasource.write.hive_style_partitioning": "true",
}
(
    changes.write.format("hudi")
    .options(**hudi_options)
    .mode("append")
    .save(enrollment_path)
)

mode("append") on the Hudi data source still means "write this batch through the Hudi operation," and the operation is upsert. That wording confuses everyone once. The Spark save mode is not the Hudi operation. An upsert on a copy-on-write table rewrites base files for affected groups. It does not append a second copy of the key when the index hits.

course_id as the partition is a choice you must defend. It helps a course-scoped read and hurts a global "all unenrollments in the last hour" scan, and a hot course becomes a large partition. If the workload is incremental by time, partition by a coarse date of the change, or use a non-partitioned layout if the table is small enough that lesson 10's current-state table fits a single merge in Iceberg anyway. Hudi partitioning is still a path. High cardinality still hurts. learner_id is still the wrong partition.

Multi-writer Hudi needs a lock provider. The project documents lock providers, including one based on DynamoDB for AWS. A single writer does not need one. Two Spark jobs upserting enrollment_current without a lock can corrupt the timeline. Decide the writer count before you pick merge-on-read, because compaction is yet another writer that must lock against ingestion.

Cleaning: configure retained commits so an incremental consumer that was down overnight can still read the instants it missed. The default that deletes aggressively will strand that consumer the way a short Delta vacuum strands a stream. Read the cleaner retention settings for your version and set them to the consumer's recovery window. Savepoint the published instant in addition, if completion were ever served from Hudi.

What not to copy between formats

Iceberg hidden partitions do not exist in Hudi just because both are "lakehouse." Hudi precombine does not exist in Iceberg; Iceberg MERGE uses the SQL you wrote, so the ordering rule is the WHEN MATCHED clause, which you can see. Delta txn versions do not automatically apply to Iceberg writes. When you port the purchase merge from lesson 50 to Iceberg, you port the match key and the explicit column list, and you replace vacuum with expire_snapshots. You re-test the retry. You do not assume the txnAppId option is recognized.

Practical Implementation or Design Walkthrough

Problem

Select a format for two grains and leave a paper trail a reviewer can reject. Grain A is completion-v3, daily replace, Spark writes, a Trino or Snowflake reader must see the same snapshot. Grain B is enrollment current state, keyed by enrollment_id, updates throughout the day, one Spark writer.

Decision

Grain A is Iceberg on a REST or Glue catalog. The multi-engine requirement is the deciding row. Delta would force the SQL engine onto a runtime that reads _delta_log at the protocol you enable. That can be right inside Databricks. It is not this grain's constraint once a second engine is in the requirement. Hudi is not indicated: the write is a daily merge, not a log-structured upsert.

Grain B is Hudi copy-on-write if incremental consumers and the index are requirements you will staff. If the only reader is a batch merge every fifteen minutes and the table fits, Iceberg MERGE on enrollment_id keeps one format in the company. The walkthrough records that alternative and picks Hudi only with the incremental consumer named. A fictional consumer does not count.

Steps for grain A

1. Create the table with days(event_time) and the lesson 10 grain, plus definition_version.
2. Load Tuesday through a merge or an overwrite of the partition transform's files via the Iceberg writer, not via a raw Parquet drop.
3. Run the lesson 17 gate against the new snapshot id from the snapshots metadata table.
4. Publish that id. Tag it if you want expiry to refuse to drop it.
5. Start a reader on the previous snapshot, commit the next day, show the pinned reader unchanged.
6. Run rewrite_data_files on a closed month in a lab with small files. Show a new snapshot and unchanged counts. Show publication unmoved.
7. Attempt to point a second engine at the Hadoop catalog path and record why you will not. The production config uses type=rest or type=glue.

Steps for grain B

1. Define change_sequence as a long that increases for every change of enrollment_id, including replays you assign new sequences to only if they are newer. Write the rule down.
2. Upsert three changes for one key out of order in event time and in the correct order of change_sequence. The latest sequence remains.
3. Repeat with precombine set to event_time and a correction that moves event time backwards. Show the correction lost. That demonstration is the point of the exercise.
4. If you chose merge-on-read, run one compaction and show a read of the base file. If you chose copy-on-write, show that an update rewrote a base file and that the timeline has a completed commit.
5. State the single-writer or the lock provider. Do not leave it implied.

Expected evidence

A one-page decision, the snapshot id or instant time on a fake publication row, the failed precombine demonstration, and a catalog type that is not hadoop for grain A.

Enterprise Real-World Scenarios

The version hint that lost Tuesday

Two Spark jobs write the completion Iceberg table on S3 using the Hadoop catalog, because a local tutorial used type=hadoop and the bucket already existed. Each job reads a version hint, writes a metadata file, and puts a new hint. The puts race. Monday's reader sometimes sees a snapshot that does not include the late-day merge, and a retry "fixes" it an hour later without a publication change. The counts move with no definition_version change. The files are all valid Iceberg metadata. The pointer is not a compare-and-swap.

The repair moves the table to the Glue catalog or to the internal REST catalog, using Iceberg's documented register or migrate procedure for the version you run, rehearsed on a copy. Jobs lose direct access to overwrite the hint. The publication row stores snapshot_id from the catalog's current snapshot after the gate, not from a hint file a human can list. The post-incident check is a config test that fails the pipeline if catalog.type is hadoop outside the lab profile. The data did not need a new format. It needed the commit sequence from this lesson's diagram. Delta on S3 had the same class of bug in lesson 50 when the log store was missing. The lesson is the pointer, not the vendor.

The correction that precombine threw away

Support fixes an enrollment row: the learner was unenrolled, then the unenrollment is reversed, and the reversal event is produced with the original business event_time because the upstream system stamps the enrollment's start time on every change. The Hudi table's precombine field is event_time. The reversal's timestamp is older than the unenrollment's timestamp. Copy-on-write upsert keeps the unenrollment. The cohort API, which reads this table as current state, still shows the learner out. The CDC topic has the reversal. The table does not.

The repair sets precombine to change_sequence from the Debezium LSN or an equivalent monotonic field on the event, backfills the key from the topic while retention allows, and adds a test that applies an older event_time with a newer sequence and expects the newer sequence to win. The cohort's freshness SLO from lesson 47 is innocent. The fold was fresh and wrong. Ordering fields are part of correctness, the way the match key of a Delta merge is part of correctness. A timestamp is a measure of when something happened in the world. It is a precombine field only when the source promises it increases for every successive state of the key.

Best Practices

• Put Iceberg tables in a catalog with a conditional pointer update. Keep the Hadoop catalog in single-process labs.
• Filter on the transformed column, such as event_time with days(event_time). Do not maintain a second date column that can drift.
• Evolve schema by field id through the table API. Refuse directory drops of Parquet as a load path.
• Schedule rewrite_data_files when small files or delete files show up in the plan. Expire snapshots on the audit window, and pin published snapshots with a tag if expiry would otherwise eat them.
• Treat equality-delete growth as a maintenance signal, not as a cheap upsert you can ignore for a month.
• Give Hudi a primary-key record key and a monotonic precombine. Demonstrate a backwards business timestamp in a test.
• Staff compaction and cleaning on merge-on-read, or choose copy-on-write and pay the write amplification knowingly.
• Configure a lock when a second Hudi writer, including compaction on another cluster, exists.
• Size cleaner retention for the incremental consumer's downtime.
• One grain, one format. Use the comparison table. Name the engine that is not allowed to be surprised.
• Keep the publication row. Store an Iceberg snapshot id or a Hudi instant. Do not publish "latest."
• Port match keys and quality gates when you leave Delta. Do not port config keys that the new format does not define.

Common Mistakes and Anti-Patterns

Hadoop catalog on a shared bucket. The pointer race loses commits. Use REST, Glue, Hive, or JDBC.

A date column plus a hidden day partition that disagree. You will debug a Tuesday that is empty in one filter and full in the other. One source column.

Directory loads. Field ids never get assigned correctly for those files, or they get assigned as a blind add that will not match the next evolution. Use the writer.

Expiry with retain_last 1 and older_than of yesterday, on a table with an audit promise. You kept one snapshot and deleted the audit. Set the window first. Then choose retain_last.

Equality deletes as the steady state. Reads apply a longer chain every day. Compact. If the update rate is the point of the table, Hudi merge-on-read or a designed Iceberg compaction schedule belongs in the design, not in the incident.

Precombine on event_time for mutable state. Corrections that reuse an earlier business time disappear. Use a sequence.

mode("append") understood as "add another row for this key." On a Hudi upsert, append is the Spark save mode. The operation is the option. Read the option.

Two writers without a Hudi lock. Timeline corruption is not a retry. It is a restore from a savepoint, if you have one.

Partition by learner_id because hidden partitioning "solved cardinality." It did not. Bucket transforms still create many tasks and many files if you choose a huge bucket count. Partition for the query.

Running Delta, Iceberg, and Hudi on completion so each team keeps its habit. Three maintenance stacks, one metric. Pick.

Assuming Snowflake features on an Iceberg table you have not tested. Time travel, cloning, and clustering in lesson 53 are described for native tables. Verify on Iceberg before you promise them.

Troubleshooting and Design Review

Symptom: Spark and Trino return different rows for "the current table."

Likely cause: they are not using the same catalog, or one cached a metadata file, or one is reading a path. A protocol or spec feature one engine skips is the next hypothesis.

Investigation: the metadata file location each engine prints, the snapshot id, and the catalog URI or Glue database.

Corrective action: one catalog configuration, checked into the job and into the SQL engine's catalog definition. Refresh metadata. If a feature is unsupported, stop writing it.

Symptom: Queries get slower every day. Snapshots are small. File counts in manifests climb, many of them delete files.

Likely cause: merge-on-read with equality or position deletes and no rewrite.

Investigation: the manifests summary, delete-file counts, and whether rewrite_data_files is scheduled.

Corrective action: rewrite, then expire snapshots that only the uncompacted tree needed, inside the audit policy. Confirm counts against the published snapshot before and after. Publication stays on a snapshot whose counts you checked, or moves only through the gate if the rewrite is value-preserving and your policy allows either id.

Symptom: An enrollment key shows an old state. The topic has a newer change.

Likely cause: precombine dropped it, or the index wrote a duplicate file group and the read picks one, or the incremental consumer skipped an instant.

Investigation: the timeline for the key's partition, the precombine field, a direct query by record key, and the consumer's stored instant.

Corrective action: fix ordering, backfill the key, add the regression test. If you find duplicate file groups, follow Hudi's repair guidance for the version you run rather than deleting Parquet by hand. Hand-deleting base files is the directory-list bug again.

Symptom: expire_snapshots deleted a snapshot the auditor asked for, and the publication row still names it.

Likely cause: expiry did not know about the publication row, and no tag pinned the snapshot.

Investigation: the publication table versus SELECT * FROM table.snapshots. Whether a tag exists.

Corrective action: rebuild from raw if the files are gone. Pin the published snapshot with a tag or a branch the expiry procedure documents as retained. Make the expiry job read the minimum snapshot the publication table still references and refuse to expire it. The row in your database is not visible to Iceberg unless you also pin inside Iceberg.

Symptom: Hudi incremental output missed a morning of changes after a long pause.

Likely cause: cleaner retention was shorter than the pause, so the start instant no longer exists.

Investigation: the timeline, the cleaner policy, the consumer's start instant.

Corrective action: increase retained commits or hours to the recovery window. Re-bootstrap the consumer from a savepoint or a full snapshot, idempotently. Do not advance the consumer's stored instant past a gap and call it caught up.

A design review asks: which catalog holds the pointer? What is the snapshot id or instant on the publication row? What transform or partition path matches the restatement filter? What orders two writes of the same key? Who runs compaction, and what pin stops expiry from eating Monday? Which engines read this table, and which protocol features are they certified for? Why is this not the other format?

Interview Questions and Answers

Q1. What does an Iceberg reader open, in order?

The catalog gives it the current metadata file, or the metadata file for the snapshot it asked for. That file yields the snapshot. The snapshot yields a manifest list. The manifest list yields manifests. Manifests yield data files and delete files, with statistics the planner uses to skip files. The reader does not list the warehouse prefix and open every Parquet object. If it does, it is not using the table.

Q2. Why can Iceberg rename a column without rewriting Parquet?

Columns have field ids in table metadata. The name is a label on the id. Files written under the old name still carry the id. A reader with the new schema maps the id to the new name. A Hive-style reader that matches on the Parquet column name would see a null. I want the id behavior on any table whose schema will evolve, and I only get it if writers use the Iceberg API.

Q3. What is hidden partitioning, and what bug does it remove?

The partition transform is stored in the spec, and planners prune from the real column. PARTITIONED BY (days(event_time)) means a filter on event_time can skip days without a writer-maintained event_date that might disagree with the timestamp. I still choose the transform for the query. Hourly transforms on a daily job are how I recreate a small-file problem inside a fancier spec.

Q4. When do you choose copy-on-write over merge-on-read in Iceberg or Hudi?

When reads dominate and I want them to be plain scans of base files, I pay the rewrite at write time. When updates are frequent and readers can tolerate, or are configured for, applying delete files or Hudi logs, I write the delta quickly and I schedule compaction. I do not pick merge-on-read and then leave compaction unowned. That is a slow read with extra steps.

Q5. Why is the Hadoop catalog disallowed on the production completion table?

It records the current metadata pointer in the filesystem. On S3 that pointer is not a compare-and-swap you can trust with two writers. Iceberg documents this catalog as a simple implementation. I use Glue, Hive, JDBC, or REST so the pointer swap is the commit. A lab on a local disk can use Hadoop. The production config test should reject it.

Q6. What must be true of a Hudi precombine field?

For a given record key, the field must order versions the way the business orders truth. The greater value wins. A business timestamp that a correction can move backwards will keep the wrong row. I use a monotonic change sequence from the log. I test an out-of-order pair. I do not copy event_time into the option because it was convenient in the schema.

Q7. How does Hudi's operation relate to Spark's save mode?

mode("append") means Spark will call the data source with the batch. hoodie.datasource.write.operation set to upsert means Hudi applies those rows as an upsert against the record key. Append mode is not "insert another copy." If I wanted blind inserts I would set the operation to insert, and I would not call the table current state.

Q8. How do you keep snapshot expiry from deleting Monday?

I set older_than to the audit window and retain_last high enough to cover retries inside that window. I tag or branch the published snapshot if the procedure will otherwise treat it as expired, and I make the expiry job refuse any snapshot id still stored on the publication row. The publication row alone is invisible to expire_snapshots. The pin has to be something Iceberg knows about, or the job has to read the row and pass snapshot ids carefully. I do not pass the current snapshot id to a manual expire list. The docs warn that expiring an explicit id must not target the current snapshot you still serve.

Q9. How do a Delta version, an Iceberg snapshot, and a Hudi instant compare as publication handles?

Each is the token a reader pins. Delta's is a monotonic integer in _delta_log. Iceberg's is a snapshot id reached through the catalog. Hudi's is a completed instant time on the timeline. I store that token on the lesson 49 publication row. I do not store a file path. Maintenance creates new tokens that must not become official by default. The engines differ. The serving rule does not.

Q10. Enrollment current state is a few million keys, updated during the day, and the only reader is a 15-minute batch. Would you introduce Hudi?

Not by default. I would MERGE into the Iceberg or Delta table I already operate, on enrollment_id, with the change sequence in the WHEN MATCHED AND s.change_sequence > t.change_sequence clause so ordering is visible SQL. I would introduce Hudi if I had measured that merge's write amplification or if an incremental consumer needed instants. A new format has a timeline, an index, and a cleaner to staff. The key and the ordering rule are the requirements. Hudi is one implementation.

Q11. A teammate wants completion in Iceberg, a replica in Delta, and the cohort in Hudi, three teams, three habits. What do you allow?

Completion in Iceberg if that is the multi-engine server, and the replica is a copy of the published snapshot into whatever the replica engine requires, without a second definition. The cohort can stay on the engine that already serves it. I do not add Delta as another writable completion. I do not move the cohort to Hudi in the same quarter as the Iceberg migration unless the upsert problem is the actual incident. One migration, one new on-call skill, a publication token that stays stable in meaning.

Hands-On Exercises

Exercise: Pick a format and break the wrong ordering field

Problem statement. Choose Iceberg or Hudi for completion and for enrollment current state. Implement the Iceberg pointer correctly in config, and demonstrate a Hudi precombine mistake beside a monotonic sequence.

Tasks.

1. Fill the comparison table for both grains with the engines you actually have. One format per grain, or one format for both, with a sentence on what you refused.
2. Write the Spark catalog config for Iceberg using rest or glue. Write the lab-only Hadoop config on a separate profile and state the race you refuse to import.
3. Define the completion table with days(event_time) and the existing grain. Show a snapshot id query you would store on publication.
4. Show a rewrite_data_files and an expire_snapshots call with an older_than that matches a 30-day audit, and name the pin that protects the published snapshot.
5. Configure a Hudi upsert on enrollment_id with change_sequence. In a second run, use event_time and show a backwards correction lost.
6. State whether a second writer exists. If it does, name the lock. If it does not, say so.
7. Describe the incremental consumer's retention, or say there is no incremental consumer and Hudi is therefore optional.

Expected deliverables. The decision page, the catalog config, the expiry call, and the precombine test notes.

Evaluation rubric.

You pass when a reviewer can see which system commits, which token gets published, and which ordering field would have dropped the enrollment reversal.

Lesson Summary

Iceberg commits by swapping a catalog pointer to a metadata file whose snapshot names manifests and data files. Column ids make rename and reorder metadata operations. Hidden transforms partition on the real timestamp. Delete files speed writes and must be rewritten. Snapshot expiry is the vacuum, and it must be taught about published snapshots. The Hadoop catalog does not belong on S3 for this table.

Hudi commits instants on a timeline, upserts by record key, and keeps the greater precombine value. Copy-on-write rewrites base files. Merge-on-read defers work to compaction you must schedule. The ordering field is a correctness setting. A business timestamp that can move backwards is the wrong field.

Delta, Iceberg, and Hudi can all serve a lesson 49 publication token. They cannot all serve it for one grain without tripling the on-call. Choose from the writer, the engines, and the maintenance you will actually run. Native Snowflake remains a different server, not a fourth open format.

Databricks Lakehouse Platform is the operating environment when Delta and a managed workspace are the choice you just made. Snowflake Architecture and Engineering is the environment when the server is a warehouse and the lake is the ingest.

Further Reading

• Iceberg table spec — metadata files, snapshots, manifests, partition transforms, delete files, sequence numbers.
• Iceberg Spark procedures — expire_snapshots, rewrite_data_files, rewrite_manifests, and named arguments.
• Iceberg catalogs — hive, hadoop, rest, glue, jdbc. Read the Hadoop catalog's production warning before you copy a tutorial.
• Hudi overview — timeline, table types, indexing, and the writer model.
• Hudi write operations — upsert, insert, and the record key. Confirm option keys for your version.
• Delta Lake Architecture and Operations — the format this lesson compared and did not re-specify.
• Data Lakehouse Architecture — publication, one server, maintenance versus official versions.
• Snowflake Architecture and Engineering — native tables and the limit of assuming Iceberg parity.