Databricks Lakehouse Platform

Data Engineering & Modern Data Platforms · Lesson 52 · Modern Data Platforms and Cloud

Lesson Overview

The Delta log from Delta Lake Architecture and Operations can live on any Spark runtime that speaks it. TechLearningPro still needs an operating platform: a place jobs run as a known identity, a catalog that grants SELECT on completion and withholds raw, a SQL surface for the studio's analysts, and a cluster that exits when the publish ends. Databricks is that platform when the decision in Apache Iceberg and Hudi Fundamentals was "Delta, one engine family, we will staff this workspace."

This lesson designs the workspace around the publication path you already specified. Unity Catalog is the three-level namespace and the grant boundary. Job clusters run the commit. SQL warehouses run interactive SQL. All-purpose clusters are a scratch environment with an auto-termination clock, not the production publisher. Storage credentials and external locations are how the workspace reaches s3://tlp-raw without a personal key in a notebook. The Jobs scheduler is the single writer lesson 48 wanted from Airflow, or it is a task Airflow triggers. Either way, one schedule owns the merge.

The lesson does not invent REST payloads. Cluster policies, grants, and job settings are described as the product documents them, and anything you export from your workspace is the schema you review. Pricing changes. DBUs are named as the billing unit, not quoted as a rate. Photon is the vectorized engine in Databricks Runtime, not a multiplier you can put in a design doc without a measurement.

Snowflake remains the alternative server in the next lesson. A replica there still carries source_snapshot_id. This workspace does not get a second completion definition beside it.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate the workspace, the Unity Catalog metastore, and the cloud storage the tables point at.
2. Grant catalog privileges so the studio can read published completion and cannot read raw or landing.
3. Choose managed or external Delta tables without attaching the live Connect prefix.
4. Run production publishes on job clusters and analyst SQL on SQL warehouses, and say what an all-purpose cluster is allowed to be.
5. Structure a job as tasks with a retry that is safe because the merge is idempotent.
6. Point storage access at a storage credential and an external location, not at a key in a notebook.
7. Keep OPTIMIZE and the quality gate as tasks that do not silently move the publication row.
8. Read a DBU bill as a consequence of SKU, cluster lifetime, and warehouse size, using the vendor's current rate card rather than a remembered price.

Prerequisites

• Lesson 13: zones, ELT, and one curated job. The workspace is where that job runs when Spark no longer fits a single host.
• Lesson 20: schedules, retries, and a single active run. A Databricks job is an orchestrator with the same duties, or a worker an orchestrator calls.
• Lesson 23: pipeline tests. A notebook that passed yesterday on an interactive cluster is not a test.
• Lesson 47: prefix clock versus table clock. The job's commit time is the table clock.
• Lesson 48: closed hours, markers, and the prohibition on listing an open hour as a version.
• Lesson 49: publication row, one server, maintenance distinct from official.
• Lesson 50: MERGE, versions, vacuum retention, log store on S3, schema enforcement.
• Lesson 51: the choice that put this grain on Delta. If the choice was Iceberg with several engines, do not force the grain into this workspace to make the lesson feel used.

Detailed Core Concepts

What you are buying

A Databricks workspace is a control plane: notebooks, jobs, clusters, SQL warehouses, and identity. It is not the data. The data sits in object storage. Unity Catalog, when you use it, is an account-level metastore in a region, shared by the workspaces you assign to it. The metastore maps catalog.schema.table to a storage location and enforces grants. Cloud services still bill the storage account. Databricks bills compute in Databricks Units, DBUs, by SKU. The SKU for an all-purpose cluster and the SKU for a job cluster are not the same product. The current price list is the number. This lesson refuses to print a price that will be stale.

Databricks Runtime is Spark plus Delta plus libraries, on a version line you pin. Pinning matters because a protocol feature enabled on a newer runtime can make a job on an older runtime fail, which lesson 50 already saw. Production jobs name the runtime. They do not float to "latest" on the morning of a launch.

Photon is an execution engine inside that runtime that runs a subset of Spark SQL in a vectorized way. Enable it when a measured SQL workload benefits, on a runtime that documents it. Do not write "Photon makes the publish ten times faster" in the design. Measure the publish you have.

Three levels, and the grants that match the zones

Unity Catalog names are catalog.schema.table. A practical split for this course:

USE CATALOG and USE SCHEMA are required before SELECT does anything useful. A grant of SELECT on one table does not imply the right to browse every other table in the schema. Do not grant ALL PRIVILEGES on the catalog to the analyst group because the UI felt empty. That grant includes the right to create and to read raw.

sqlCopy
GRANT USE CATALOG ON CATALOG tlp TO studio_readers;
GRANT USE SCHEMA ON SCHEMA tlp.learning TO studio_readers;
GRANT SELECT ON TABLE tlp.learning.completion TO studio_readers;
GRANT SELECT ON TABLE tlp.learning.publication TO studio_readers;

The job's service principal gets MODIFY and SELECT on the tables it writes, and USE on the schemas. It does not get ALL PRIVILEGES on tlp.raw if a narrower SELECT plus MODIFY on the one target is enough. Landing never becomes a schema. Partner files with a column the boundary must drop stay in s3://tlp-landing with no external location granted to humans.

Group names in your account may need backticks if they contain characters the SQL parser reserves. The privilege names above are the Unity Catalog privileges documented for securables of this type. If a future runtime splits a privilege, the migration note in the SQL reference wins over this page.

Managed storage, external locations, and the prefix you must not register

A managed table stores its files under a location Unity Catalog controls for that catalog or schema. DROP TABLE drops the metadata and the files. Use managed tables for learning.completion when you want the catalog to be the lifecycle, and set delta.deletedFileRetentionDuration anyway, because drop is not the same as vacuum of old versions.

An external table points at a path you manage. DROP TABLE drops the catalog entry and leaves the files. Use an external table when a path has to live beside another system, and accept that bucket lifecycle can now destroy a Delta log the catalog still believes in. Lesson 50 forbade lifecycle expiry on Delta prefixes. That rule is louder for external tables.

Do not create an external table whose location is the live Connect prefix s3://tlp-raw/learning/progress/. Connect is not writing _delta_log. A CONVERT TO DELTA in place, while the sink still puts objects, races the open hour. Copy or COPY INTO a closed run prefix, the one the lesson 48 marker names, into a table the job owns. The marker remains the ingest clock. The Delta version remains the table clock.

An external location is a securable that binds a URL to a storage credential. The credential is the IAM role Unity Catalog assumes. Databricks documents creating the credential, then the location:

sqlCopy
CREATE EXTERNAL LOCATION IF NOT EXISTS tlp_raw_loc
URL 's3://tlp-raw/'
WITH (STORAGE CREDENTIAL tlp_raw_cred)
COMMENT 'Raw prefixes the commit job may read. Not landing.';

Create tlp_raw_cred with the statement in the current Unity Catalog SQL reference that registers the IAM role ARN. On AWS, the role has to trust the Databricks principal and, in current documentation, be self-assuming. The trust policy's external id comes from the credential after you create it. Follow that page's order. Do not paste a trust policy from a blog that predates the self-assuming requirement.

CREATE EXTERNAL LOCATION does not by itself let every user read the bucket. Grants do. READ FILES on an external location is a powerful grant: it reads paths under that URL, which is how a curious notebook bypasses table grants if the location is the whole bucket. Scope the location URL as tightly as the product allows, and grant READ FILES to the job principal, not to studio_readers. Studio reads tables. The job reads closed prefixes and writes tables.

A classic cluster instance profile that can GetObject on the bucket is the older path. With Unity Catalog it is also a bypass, because the profile's permission is not the user's grant. Production job clusters that run Unity Catalog tasks should not depend on a wide instance profile for table data. Admin instance profiles for cluster logs are a separate, narrow concern.

Clusters

An all-purpose cluster is an interactive cluster. It stays up until idle termination or until someone forgets termination. It bills the all-purpose SKU the whole time. It is appropriate for exploration, with a termination window measured in tens of minutes, a cluster policy that caps the node type, and no production schedule pointed at it.

A job cluster is created for a job run and terminated at the end. It bills the jobs SKU for the run. The publish uses a job cluster. The runtime and the worker count are in the job definition, reviewed like code. A job that "attaches to the shared interactive cluster" fails when an analyst restarts that cluster, and it hides the publish's real cost inside someone's notebook session.

Cluster policies are the guardrail. A policy can restrict runtime versions, require auto-termination on all-purpose clusters, and cap autoscale. Policies are how you stop a 6-node all-purpose cluster from becoming the default click. They are account configuration, owned by the platform group, not a wiki request.

SQL warehouses are the compute for Databricks SQL. They are not Spark clusters you ssh to. Serverless SQL warehouses, where your cloud and account offer them, start quickly and bill on a SQL SKU. Classic SQL warehouses use clusters you size. Point analyst queries and the studio's read-only SQL at a warehouse. Point the MERGE at a job cluster. Mixing them means the publish's shuffle and a dashboard's scan compete, and the failure looks like a slow catalog.

Jobs

A job is a graph of tasks. Each task has a type the product documents: a notebook in Git, a Python file, a SQL task on a warehouse, a dbt task, and others. Tasks declare dependencies. The completion publish is a short graph:

1. Resolve closed markers and dead-letter completeness for the window. Fail the task if the hour is open or the purchase gap is non-zero.
2. MERGE into the Delta table on a job cluster. Conflict retry is the task retry. The merge itself is idempotent, from lesson 50.
3. Run the lesson 17 gate against the version the merge committed.
4. Insert the publication row only if the gate passed. This task is the only one allowed to write learning.publication.
5. OPTIMIZE closed partitions on a schedule that is a different job, not a step that writes publication.

Set a maximum of one concurrent run for the publish job so two windows do not rewrite the same partition. Retries cover infrastructure failure. They are safe only because step 2 merges and step 4 writes a specific version, not "whatever is current" at the moment the task woke up. If step 2 committed and step 4 did not run, the retry must publish that same version after re-checking the gate, not merge a second copy. The lesson 50 txn pattern, or a merge plus a publication insert keyed by load_id, is the mechanism. Test the kill between step 2 and step 4. That kill is the drill.

Schedule in UTC with the cron the Jobs service documents. Alert the table owner, not the Connect on-call. The sink can be healthy while step 3 fails.

Deploy the job from Git. Databricks Repos or Git folders, and Databricks Asset Bundles or the Jobs API, are the documented ways to make a workspace match a commit. A notebook saved only in the workspace is an unreviewed production path. The bundle YAML in your repo should be the YAML your installed CLI validates. This lesson does not freeze a bundle schema that your CLI version might reject. It freezes the task graph and the identity.

The job's identity is a service principal. A personal access token in a notebook or in a CI variable named after an engineer is a standing credential. Rotate principals by changing the job's run-as, not by rewriting cells.

Where Airflow still fits

Apache Airflow Architecture and DAGs may already schedule the morning gate. You can keep Airflow as the owner of "Monday's window" and have it trigger this job, or you can let the Databricks job be the owner and retire the duplicate cron. Two crons that both merge Tuesday are the conflict lesson 50 described, now with a vendor boundary. One owner. If Airflow triggers the job, Airflow waits for the run result. It does not also run a local Spark submit against the same table.

SQL the warehouse is allowed to run

Analysts and the API's read model use the published version:

sqlCopy
SELECT c.course_id, count(*) AS completions
FROM tlp.learning.completion VERSION AS OF 88 AS c
INNER JOIN tlp.learning.publication AS p
  ON p.dataset = 'learning.completion'
 AND p.definition_version = 'completion-v3'
 AND p.snapshot_version = 88
WHERE c.definition_version = 'completion-v3'
GROUP BY c.course_id;

The join is redundant if the application already substituted the version from publication. Redundant is better than a dashboard that hard-codes VERSION AS OF 88 until someone edits it, or a dashboard that omits the version and reads the head. A view that always joins to the latest publication row is the supported interface. Grant SELECT on the view. Do not grant MODIFY. The view's definition is code.

COPY INTO loads files from a closed prefix into a Delta table and, as documented, tracks files so a repeated copy does not duplicate them. Use it for the raw append path when the files are closed and the schema matches. Confirm options in the COPY INTO reference for your runtime, including how to force a reload. Do not COPY INTO the parent hour prefix. The marker's run prefix is the FROM path. mergeSchema stays off.

Cost, without a fake rate card

You pay DBUs for the minutes a job cluster exists, DBUs for the SQL warehouse while it is running or serverless-billed, and cloud VM charges where the SKU is not serverless. You pay the cloud for the bytes. A Saturday all-purpose cluster is a DBU line with no publish attached. Auto-termination is the control. A warehouse with a multi-hour idle timeout is the same leak on the SQL SKU. Set the timeout to the interactive gap you actually have, often a few minutes for a warehouse that auto-resumes.

The publish's cost is roughly the scan of the restatement window plus the write. Narrow the window. Do not scan a year to replace Tuesday. Lesson 13 already scoped the replace. The platform will not remind you.

Photon, autoscaling, and a larger node type are experiments with a before and after on bytes and minutes. They are not the first response to a publish that is late because the gate is failing. A failing gate on a bigger cluster fails faster and costs more.

Architecture and Design Diagrams

mermaidCopy
flowchart TB
    subgraph ident [Identity]
        sp[Publish service principal]
        readers[studio_readers]
    end

    subgraph uc [Unity Catalog]
        rawS[tlp.raw]
        comp[tlp.learning.completion]
        pub[tlp.learning.publication]
        loc[External location tlp_raw_loc]
    end

    subgraph compute [Compute]
        jobC[Job cluster]
        wh[SQL warehouse]
        allp[All-purpose cluster with termination]
    end

    subgraph store [Object storage]
        prefix[Closed run prefix]
        files[Managed Delta files]
    end

    sp --> jobC
    jobC --> loc
    loc --> prefix
    jobC --> comp
    jobC --> pub
    readers --> wh
    wh --> pub
    wh --> comp
    allp -.-> rawS
    jobC --> files

The dotted edge is break-glass, not the dashboard. Readers reach tables through the warehouse. They do not receive READ FILES on the raw location.

mermaidCopy
sequenceDiagram
    participant Sched as Job schedule
    participant Gate as Completeness task
    participant Merge as MERGE task
    participant Pub as Publication task
    participant Opt as Separate OPTIMIZE job

    Sched->>Gate: Closed markers only
    alt Hour open or dead-letter gap
        Gate-->>Sched: Fail, do not merge
    else Window complete
        Gate->>Merge: Run id and prefix
        Merge->>Merge: Commit Delta version V
        Merge->>Pub: Pass V to the gate SQL
        alt Quality gate passes
            Pub->>Pub: Insert publication for V
        else Gate fails
            Pub->>Pub: Leave the previous publication
        end
    end
    Note over Opt: May commit V+1 later and must not insert publication

One concurrent run. The kill you drill is between the merge commit and the publication insert.

Technical Explanation

Identity and the notebook that still has a key

Workspace users authenticate as themselves. Jobs run as a service principal you assign. Unity Catalog evaluates the principal that runs the query. A notebook cell that sets an AWS key in the session to read s3://tlp-landing is outside that evaluation. Secret scopes exist so a legitimate external call can retrieve a secret without pasting it. They are not a reason to store a bucket key the external location was meant to replace. The review check is a search of the repo and the workspace for access-key strings, plus a grant review: humans do not have READ FILES on tlp-landing, and there is no external location on that bucket for them.

Audit logs for Unity Catalog, which the platform documents, answer who selected the table. They do not answer who downloaded a file with a personal key. That gap is why the key is forbidden.

The publication view

sqlCopy
CREATE OR REPLACE VIEW tlp.learning.completion_current AS
SELECT c.*
FROM tlp.learning.publication AS p
JOIN tlp.learning.completion VERSION AS OF p.snapshot_version AS c
  ON c.definition_version = p.definition_version
WHERE p.dataset = 'learning.completion'
  AND p.definition_version = 'completion-v3'
  AND p.published_at = (
    SELECT max(published_at)
    FROM tlp.learning.publication
    WHERE dataset = 'learning.completion'
      AND definition_version = 'completion-v3'
  );

If your runtime rejects VERSION AS OF with a column reference, resolve the version in a task or a function the SQL reference allows, and keep the rule: the view does not scan the head. Check the Databricks SQL manual for the time-travel syntax your warehouse version accepts. The rule is portable even when the syntax needs a parameter. A view that omits time travel and filters definition_version only will show uncommitted repairs that share the version string. The snapshot integer is the pin.

What a cluster policy should refuse

Refuse all-purpose clusters without auto-termination. Refuse node types you do not pay for on purpose. Refuse a runtime older than the protocol your tables require, and refuse an arbitrary newer runtime on the production policy until you have run the reader matrix. Allow a separate "explore" policy with small nodes and a short timeout. Production job definitions use the production policy. An analyst who needs a large interactive cluster files a change. The Friday export that became a 20-worker cluster is how the DBU graph spikes with no commit in history.

Git and parameters

Job parameters carry the event_date window and the definition_version. The notebook or Python file reads parameters the Jobs runtime documents (dbutils.widgets for notebooks, arguments for Python tasks). A cell that sets event_date = "2026-11-10" and was edited in the workspace UI will republish Tuesday forever. Parameters come from the schedule or the trigger. The default, if someone runs the job by hand, should be "yesterday UTC" computed in code, or no default and a required parameter. A silent default of a hard-coded launch day is a repeat incident.

Lineage in the catalog

Unity Catalog can record lineage for tables it governs. Use it as a hint. The publication row remains the audit token, because lineage that says "this table was read" does not say "Monday served version 88." Data Lineage and Metadata Management asked for a sentence a reviewer can reproduce. The row is that sentence. The catalog graph is optional support.

Practical Implementation or Design Walkthrough

Problem

Stand up the least workspace that publishes completion-v3 from closed progress hours, serves it to studio_readers, and does not leave an interactive cluster as the publisher.

Steps

1. Assign the workspace to the Unity Catalog metastore. Create catalog tlp and schemas raw, learning, commerce. Set a managed storage location for tlp.learning through an external location dedicated to managed files, following the managed-storage page. Do not reuse the Connect bucket as managed storage.
2. Create the storage credential and tlp_raw_loc on s3://tlp-raw/. Grant the job principal the privileges required to read closed prefixes. Do not grant that location to studio_readers.
3. Create learning.completion as a managed Delta table, partitioned by event_date, schema from lesson 50's explicit columns, definition_version included. Create learning.publication with the columns from lesson 49.
4. Create group studio_readers and the grants in the section above. Create service principal completion_job. Grant it modify on the two learning tables and select on the raw inputs it must read.
5. Define the job with the five-step graph, max concurrent runs 1, retry on the merge task, run-as completion_job, job cluster on a pinned runtime, Git source at a commit. Parameter event_date required.
6. Define a SQL warehouse with a short auto-stop. Create the current view if the syntax check passes. Grant select on the view to studio_readers.
7. Define a second job, completion_optimize, on a cron outside the publish, OPTIMIZE of partitions older than the open window, no publication insert. Vacuum on a third schedule at the audit retention.
8. Drill: run the publish for a lab day; kill after the merge; rerun; show one row per grain and one official version. Stop the all-purpose policy's cluster and show the job still runs.
9. Search the workspace for keys and for a notebook task whose source is not Git.

Expected evidence

A grant list, a job run URL or id from the lab, history of the Delta table showing merge distinct from optimize, publication pointing at the merge version, and a warehouse query that fails when run as a user without raw access and succeeds on the view.

Enterprise Real-World Scenarios

The interactive cluster that was production

The publish notebook lives on an all-purpose cluster named shared-prod because that was the fastest way to see logs. Analysts use the same cluster for ad hoc joins. Auto-termination is off so the morning job can attach. Over a long weekend the cluster stays up. The DBU line is larger than the month's publishes. On Monday an analyst restarts the cluster to pick up a library. The scheduled run attaches mid-restart and fails. A retry starts while the first attempt's merge is committing. Concurrent runs were left at the default. The conflict retry path appends, because this notebook is the old code from the lesson 50 incident, copied forward.

The repair splits compute. The job runs on a job cluster as completion_job, one concurrent run, Git source, merge only. Analysts get a SQL warehouse and the view. The all-purpose cluster gets a policy with auto-termination and loses its instance profile's access to the table bucket. The weekend bill disappears with the idle cluster. The restart no longer kills Monday. The append fallback is deleted in the same pull request. The platform did not cause the duplicate rows. It made the unsafe notebook convenient enough to become the publisher. Convenience is the design smell.

READ FILES on the bucket root

A platform admin creates one external location, s3://tlp/, covering landing, raw, and curated, and grants READ FILES to engineers so a migration notebook can reach everything. An engineer previews a partner CSV to "see the header" and the preview includes email, which lesson 13's boundary was built to keep out of raw. The preview is in the notebook's stored output. Unity Catalog table grants on tlp.learning are perfect and irrelevant. The location grant is the door.

The repair deletes the root location. Landing has no external location in this workspace. Raw's location is s3://tlp-raw/ granted to completion_job only. A new header check runs in the boundary job, not in a notebook preview. Stored notebook outputs on the old notebook are cleared. The grant review becomes a quarterly control: any READ FILES grant must name a group that is not an analyst group and a URL that is not a bucket root. The incident is the lesson 13 email incident again, with a Unity Catalog object as the leak. Catalogs do not infer sensitivity. The URL scope does.

Best Practices

• Pin the Databricks Runtime on production jobs. Treat protocol changes as a reader-matrix change from lesson 50.
• Publish from a job cluster running as a service principal. Keep all-purpose clusters small, terminated, and off the schedule.
• Serve SQL from a warehouse and a view that pins the publication version.
• Use managed tables for completion unless an external path is a real requirement. Never register the live Connect prefix as the table.
• Scope external locations. Grant READ FILES to job identities. Do not grant it on a bucket root to engineers by default.
• One concurrent publish run. Retries are safe only if merge and publication are idempotent. Drill the kill between them.
• Keep OPTIMIZE and VACUUM in jobs that cannot write learning.publication.
• Schedule in UTC. Pass the window as a parameter. Do not hard-code dates in cells.
• Store the job in Git and deploy with the documented bundle or Jobs API. Review the exported JSON when you change a task.
• Cap clusters with policies: auto-termination, node types, runtime.
• Leave landing out of the catalog. The boundary job stays in front of email.
• One cron owner. Airflow triggers this job or this job is the cron. Not both as writers.
• Read the current DBU rate card when you estimate. Compare SKUs. Terminate idle compute before you buy a larger worker type.
• Keep the replica rule: another platform copies the published version and does not reshape completion-v3.

Common Mistakes and Anti-Patterns

All-purpose cluster as the scheduler. Idle DBU spend, restarts, and shared libraries. Job clusters exist so the publish has a lifecycle.

Concurrent runs plus a non-idempotent write. The retry that overlaps a live merge is how duplicates and conflicts become data. Cap concurrency at one. Merge.

External table on the sink prefix. Open hours and missing _delta_log become "the table." Load closed run prefixes into a table you own.

READ FILES on a wide URL. Table grants look strict while the location grant leaks landing. Scope the URL.

A personal token as the job identity. The job keeps working after the human leaves, until the token expires during the publish. Use a service principal.

Notebook-only source. The cell someone edited in the UI is not in Git and cannot be reviewed. Jobs run a commit.

Hard-coded dates. The schedule republishes a memorial day. Parameters or a computed UTC window.

Two orchestrators. Airflow and a workspace cron both merge. Pick one writer.

Vacuum or optimize task that inserts into publication. Maintenance becomes Monday. Split the jobs and the grants if you must: the optimize principal has MODIFY on the fact and not on the publication table.

Floating runtime. "Latest" enables a protocol feature or changes a SQL behavior on the morning you needed a boring publish. Pin it.

Putting the DBU price in the runbook as a constant. The runbook cites the pricing page and the SKU names. The constant will be wrong quietly.

Troubleshooting and Design Review

Symptom: The morning publish failed because the cluster was restarting, and a second run is in conflict.

Likely cause: the job attaches to an interactive cluster, and concurrency is greater than one.

Investigation: the job's cluster type, the run-as principal, overlapping run ids, Delta history.

Corrective action: switch to a job cluster, set max concurrent runs to 1, repair any duplicate grain with a merge, and only then publish. Do not increase retries on the interactive cluster and call it fixed.

Symptom: Analysts can see partner email, and table grants look correct.

Likely cause: an external location or instance profile wider than the tables.

Investigation: SHOW GRANTS on external locations and storage credentials, notebook outputs, instance profiles on clusters they can start.

Corrective action: remove the wide location, rotate any exposed file, clear outputs, and restrict who can create external locations. CREATE EXTERNAL LOCATION on the metastore should be a platform privilege, not a default analyst privilege.

Symptom: The view returns rows from a version the publication table does not list.

Likely cause: the view reads the table head, or VERSION AS OF was dropped in an edit, or it pins a hard-coded integer.

Investigation: the view definition in Git and the definition actually stored in the catalog. They diverge when someone edited the catalog by hand.

Corrective action: replace the view from Git. Grant the job identity the right to replace the view. Revoke that right from analysts. Add a test that the view text contains the publication table.

Symptom: DBUs jumped and Delta history shows no extra merges.

Likely cause: idle all-purpose clusters or a warehouse that does not auto-stop.

Investigation: the cluster list with uptime, the warehouse configuration, the account usage tables or the billing console for the SKU.

Corrective action: enforce termination in policy, stop the idle cluster, shorten the warehouse timeout. Do not shrink the publish cluster until the idle line is gone. You might make Tuesday late to save a cost you misread.

Symptom: COPY INTO loaded an open hour and the gate passed uniqueness.

Likely cause: the path was the parent hour, not the marked run prefix. Uniqueness cannot see lines that have not arrived, and it cannot see dead letters.

Investigation: the FROM path in the task, the marker store, the dead-letter completeness result.

Corrective action: delete or restore the version if you have not published it. If you published it, stop serving it, reload from the closed prefix, and put the marker check back in task 1. Uniqueness of what you loaded is not completeness of what the source committed.

A design review asks: what is the run-as principal? Which cluster type publishes? What is the max concurrency? Where is the Git commit for the job? Who can READ FILES, and on which URL? Which task alone may write publication? What runtime is pinned? What happens if the process dies after the merge and before the insert? Which SKU runs on Saturday if nobody opens a notebook?

Interview Questions and Answers

Q1. What is the difference between a Databricks workspace and Unity Catalog?

The workspace is where identities run notebooks, jobs, clusters, and SQL warehouses. Unity Catalog is the metastore that names catalog.schema.table, stores grants, and connects those names to storage through managed locations or external locations. A workspace without tables governed this way is a Spark console on a bucket. I can have several workspaces bound to one metastore so grants do not drift per workspace. The data bytes are still in the cloud account, not inside the catalog process.

Q2. Why are production publishes not run on all-purpose clusters?

All-purpose clusters are interactive, they use a different DBU SKU, and they stay up if termination is off. Analysts restart them. A scheduled attach then fails or overlaps. A job cluster exists for the run and dies afterward, on the jobs SKU, with the runtime pinned in the job definition. I keep a small all-purpose policy for exploration. I do not point the cron at it.

Q3. When do you use a managed table instead of an external table?

I use a managed table when Unity Catalog should own the files and a drop should delete them, which is the right default for learning.completion. I use an external table when the path is shared with a system that is not this catalog, and I then own lifecycle myself. I never use an external table on the live Connect prefix. That prefix is not a Delta log, and the open hour is not a version. I load the closed run prefix into a table the job owns.

Q4. What does an external location grant that a table grant does not?

A table grant allows SQL on that table, through the catalog, at a snapshot. READ FILES on an external location allows reading objects under the URL, which can include prefixes that are not tables and prefixes that hold partner files. I scope URLs and I grant file access to job principals. Analysts get SELECT on a view. A root location with a wide grant repeats the landing-zone email leak.

Q5. How do you make a job retry safe?

I cap concurrent runs at one. The write is a MERGE on the grain, or a COPY INTO that tracks files, not an append. The publication insert names the version the merge returned and is itself idempotent on load_id. I drill a failure after the commit and before the insert. If the retry appends, the design is wrong, no matter how polished the cluster policy is.

Q6. Where should Airflow sit relative to Databricks jobs?

One of them is the writer schedule. Airflow can trigger the Databricks job and wait, which is reasonable when the rest of the morning DAG is already there. A second cron inside the workspace that also merges the same window is a second writer. I do not run Spark submit from Airflow against the same Delta table "as a backup path."

Q7. What do you pin, and what do you read off a price list?

I pin the runtime, the job's source commit, the table protocol features, and the audit retention. I read DBU rates and SKU names from the current pricing page when I estimate. I do not embed a dollar figure in the lesson or the runbook as if it were a protocol constant. I do record which SKU a runaway cluster used, so the next bill is diagnosable.

Q8. Why is a SQL warehouse not the place to run the MERGE?

The merge is a Spark write with a job identity, a pinned runtime, and a Delta log store configuration. Warehouses are the analyst SQL surface. Some SQL tasks can run MERGE on a warehouse in Databricks SQL, and that can be legitimate for a small statement. The completion restatement scans a large window and belongs in the job I can size and review as Spark. I do not put it on the warehouse the dashboards share, where a long write and a Monday board compete. If I do run SQL MERGE as a warehouse task, it is still the only writer, still idempotent, and still not sharing a giant interactive cluster.

Q9. A teammate enables Photon and doubles the cluster to make the publish hit its window. What do you ask for first?

The profile: is the time in the scan, the merge, the gate, or a conflict retry? The window: are we restating three days or a year? The previous run's minutes, so Photon has a before and after. I refuse a standing order to double hardware because the job missed a window once. I also check whether the job was waiting on an interactive cluster restart. That wait will not move because Photon is on.

Q10. How does the studio read completion without reading the table head?

Through a view or a query that takes snapshot_version from learning.publication for completion-v3 and time-travels to it. The studio principal has SELECT on that view, not MODIFY on the fact, and not READ FILES. A dashboard that selects the fact table with no version is a defect. Compaction may add versions. The view does not follow them until publication says so.

Q11. What is the smallest identity design you will accept?

A service principal for the job, a group for readers, grants that name schemas and tables, external locations scoped to raw and not landing, no personal access token in Git, no access keys in notebooks. Break-glass is a separate group with an expiry, not a permanent ALL PRIVILEGES on the catalog. I can explain it on one page. If I cannot, the workspace is larger than the publish.

Hands-On Exercises

Exercise: A publish job that survives the notebook

Problem statement. Design the Databricks path for completion-v3. The publisher is a job. Readers use a warehouse. An idle cluster is not part of the design. Landing is unreachable from the analyst group.

Tasks.

1. List catalogs and schemas, and map them to lesson 13 zones. State which Connect prefix is not a table.
2. Write the grants for studio_readers and completion_job. Write the external location URL. State one grant you will not give engineers.
3. Draw the task graph: completeness, merge, quality gate, publication, and a separate optimize job. Set max concurrent runs.
4. Specify job cluster versus SQL warehouse versus all-purpose, including auto-termination for the one that needs it.
5. Show how the publication version is pinned in the reader query. If VERSION AS OF cannot take a column in your runtime, show the parameter approach and cite the manual you checked.
6. Describe the kill drill between merge and publication, and the row-count result you expect on retry.
7. Name the runtime pin and the SKU you would look up on the pricing page. Do not invent a price.
8. Say where Airflow sits, or say the workspace job is the only cron.

Expected deliverables. The grant list, the task graph, the cluster choice, the drill notes, and the location scope.

Evaluation rubric.

You pass when a reviewer can see who runs the merge, who can read email-bearing files, and which task is forbidden to mark a version official.

Lesson Summary

Databricks is the control plane for a Delta serving path you already designed: Unity Catalog names and grants, managed tables for completion, external locations scoped to closed raw, job clusters for the merge, SQL warehouses for the published view, all-purpose clusters only as terminated scratch. The job runs as a service principal from Git, one run at a time, and only the publication task flips Monday. OPTIMIZE is a different job. READ FILES on a bucket root undoes the grants. DBUs follow the SKU and the lifetime of compute. The rate card is the vendor's, read when you estimate. The live Connect prefix is still not a table.

Snowflake Architecture and Engineering is the other landing: a warehouse engine that can be the server or a replica, with micro-partitions and virtual warehouses instead of a Delta log you operate yourself.

Further Reading

• Unity Catalog — metastore, the three-level namespace, and privileges.
• External locations and storage credentials on AWS — credential order, self-assuming roles, and CREATE EXTERNAL LOCATION.
• Jobs — tasks, job clusters, concurrent runs, and schedules.
• Databricks SQL warehouses — the analyst compute this lesson separated from the publish.
• COPY INTO — file tracking for closed prefixes. Confirm options for your runtime.
• Cluster policies — auto-termination and the limits that keep all-purpose clusters from becoming production.
• Delta Lake Architecture and Operations — the log this platform must not bypass.
• Snowflake Architecture and Engineering — the other server, and the replica rule when both exist.