Data Governance, Privacy and Compliance

Data Engineering & Modern Data Platforms · Lesson 61 · Production Data Engineering and Capstone

Lesson Overview

The Monday board does not need email addresses. A debug notebook did. Someone exported a day of progress "to reproduce a grain bug," including email and a free-text feedback field that learners used like a support form. The export landed in a shared drive. The grain bug was real. The shared drive was a privacy incident. Lesson 60's monitors never fired, because volume and freshness were fine. The data was fresh, complete, and unlawful to leave there.

Data Platform Security and Governance chose identities, keys, and audit scope—the cloud IAM and encryption deep dive lives there. This lesson is the complementary engineering layer: classification, retention, purpose limitation, masking, access reviews, catalogs that tell the truth, and GDPR/CCPA-style obligations expressed as pipeline and table controls. Legal writes policies. Engineering makes them true at 06:00 when forty people query a mart that should not contain what it does not need.

TechLearningPro still moves about 40 GB of compacted Parquet progress a day into a completion mart measured in tens of gigabytes. Learners are people. Progress events can carry identifiers. Enterprise customers ask about subprocessors, retention, and deletion. DataOps and Data Platform Engineering gave assets owners. Data Observability and Operational Monitoring watches freshness without pasting sample rows into Slack. This lesson decides what those assets may hold, who may read them, how long they live, and how a deletion request becomes a job—not a spreadsheet promise.

You will not get a law degree here. You will get controls: classify columns, minimize what certified tables carry, mask for non-prod, retain with a schedule, review access, prove lineage for impact, and refuse debug paths that bypass the catalog. Module 6 operates the platform; privacy is part of operations.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate security (lesson 57) from privacy/governance engineering controls without duplicating IAM deep-dives.
2. Classify TechLearningPro columns (direct identifiers, quasi-identifiers, sensitive, public) and hang classification on the catalog.
3. Apply purpose limitation so the completion mart does not carry fields the board never needed.
4. Design retention and deletion jobs that cover lake files, mart rows, backups, and extracts—not only the warehouse table.
5. Implement masking or tokenization patterns for non-production environments and for restricted columns in production reads.
6. Run access reviews tied to lesson 59 owners and lesson 57 identities.
7. Map GDPR/CCPA-style obligations (access, deletion, purpose, minimization) to concrete pipeline behaviors.
8. Use catalogs and lineage as compliance evidence and as engineer tooling, not as shelfware.
9. Keep observability and CI from becoming secondary exfiltration paths.
10. Hand the reliability and capstone lessons a platform that can state what personal data it holds and why.

Prerequisites

• Lesson 57 for identities, encryption, and audit scope; this lesson uses those primitives for privacy outcomes.
• Lessons 59 and 60 for ownership and for monitors that must not dump PII.
• Data Lineage and Metadata Management for catalog fields classification will extend.
• Data Lake Architecture and lakehouse lessons for retention on files and table snapshots.
• Schema Registry and Event Contracts for forbidding sensitive fields in the wrong topics.
• Lesson 58 so retention is also a cost control, not only a privacy control.
• Legal/privacy partner availability in a real org; engineers do not invent lawful bases alone.

Detailed Core Concepts

Governance is decisions with evidence; privacy is people in the data

Data governance is the system of accountability for data: who owns it, what it means, what quality it promises, who may use it, and how long it lives. Privacy is the subset that concerns personal data—information that relates to an identified or identifiable person. Compliance is the ability to demonstrate that policy and law are met with evidence, not with intent slides.

Lesson 57 answered: which identity may read the object, how it is encrypted, what is audited. This lesson answers: should this column exist in this table at all, for what purpose, for how long, with what masking, and how do we delete or export it when asked?

Classification that engineers can enforce

Use a small taxonomy and apply it per column (and sometimes per table when the whole payload is sensitive):

Classification lives in the contract and the catalog, not only in a wiki. CI from lesson 59 fails if a certified mart gains a direct identifier column without a privacy review checkbox and a vault pattern.

Purpose limitation and data minimization

The board needs attempts per lesson, not emails. Purpose limitation says: collect and process for stated purposes; do not reuse progress payloads for a marketing graph without a new purpose and control set. Minimization says: the certified mart projects the columns required for the purpose.

Engineering translation:

• Ingest may land a wider raw payload for a short retention if needed for replay.
• Compaction and mart publish drop or hash fields not required.
• Extracts feeding the board use the mart, never the raw identity-bearing landing.
• New consumers submit a purpose when requesting access; "I want all columns" is not a purpose.

Retention is a graph, not a table property

Deleting rows from published.completion_attempt is necessary and insufficient. Personal data also lives in:

• Raw landing objects and failed-event side tables.
• Iceberg/Delta snapshots and time travel.
• Warehouse clones, clones of clones, and time-travel windows.
• BI extracts, CSV exports, notebook outputs, ticket attachments.
• Backups and cross-region replicas (lesson 58).
• Logs that printed record payloads (forbid this).
• Streaming topic retention (already a DataOps change type).

A retention schedule names each store, the TTL, the job that enforces it, the owner, and the evidence (job success metrics). "We delete after two years" without a job is fiction.

GDPR/CCPA-style obligations as controls

Speak carefully: regulations differ, counsel interprets, and this course is not legal advice. Engineers still implement recurring patterns:

Lawful basis / notice / consent signals. If product stores a consent or preference flag, pipelines must respect it for the purposes it governs (for example, optional analytics). Do not invent consent in the warehouse that the product UI never collected.

Access / portability. A learner or customer request for "what you hold about me" needs a keyed export job from known stores, with identity verification upstream of engineering. Catalog + lineage tell you which tables to query. Ad-hoc SQL by a well-meaning engineer is how you miss a store or over-disclose.

Deletion / erasure. A delete request becomes a ticket with a verified learner_id, a checklist of stores, jobs that delete or anonymize, and a verification count. Snapshots and backups need a written policy (delete now vs delete on expiry) agreed with counsel—engineering should not silently promise instant erasure of offline tape.

Correction. If completion records are wrong, the product systems of record correct; the lake rebuilds from authoritative events or applies a controlled correction with definition_version honesty.

Minimization and retention. Covered above; regulators care, and so does lesson 58's storage meter.

Breach notification readiness. Audit logs from lesson 57, access reviews, and knowing what was in the exported file matter more than a fancy DLP logo after the shared drive incident.

Do not sell / opt-out style signals (CCPA-like). If product marks a preference, downstream marketing marts must filter. Completion board engineering should not become a covert marketing CDP.

Masking, tokenization, and anonymization (do not confuse them)

Masking hides values from readers (j***@example.com) but may be reversible by someone with the underlying table. Good for support UIs and some analyst views.

Tokenization replaces identifiers with tokens stored in a vault mapping. Pipelines join tokens only when purpose requires. Reversible under controlled access.

Hashing with a secret pepper can support joinability without cleartext; treat rainbow risks seriously; hashing email without pepper is weak.

Aggregation and k-anonymity style thinking for published external stats: do not publish a completion rate for a company cohort of three learners.

Anonymization in the strong sense is hard; most "anonymized" exports are still personal data under a cautious reading if re-identification is plausible. Prefer minimization and access control over claiming magic anonymization.

Non-prod environments get masked or synthetic data (lesson 59). Production debug uses restricted tools with audit, not copies to laptops.

Access reviews

Quarterly (or more often for vault data): list identities with read on classified tables; owners confirm each identity still needs access for a stated purpose; remove the rest; record the review. Automate the list from IAM and catalog joins. Lesson 57 built the identity model; reviews keep it honest. Service accounts need reviews too—especially notebooks and BI service users.

Catalogs that work

A useful catalog entry for published.completion_attempt:

• Owner fields from lesson 59.
• Classification per column.
• Purpose: "learning analytics completion board; definition_versioned grain."
• Retention: "N days online; then aggregate-only."
• Upstream jobs and downstream consumers.
• Certified query ids.
• Presence/absence of direct identifiers (should be absent).

Shelfware catalogs with empty owners fail audits and fail engineers. Start with certified assets; expand outward.

Complementary to lesson 57, not a rewrite

Do not re-teach KMS key policies, bucket IAM, or row-level security mechanics here beyond using them. Do teach: which tables RLS must protect because classification says so; which columns never enter the mart; which audit log gaps matter for privacy investigations; how break-glass from DataOps is logged and reviewed. If lesson 57 scoped vault GET logs tightly for cost, this lesson explains why vault access logs stay on while progress GET logs may stay off—different classification, different need.

Architecture and Design Diagrams

Personal data should narrow as it moves toward the board.

mermaidCopy
flowchart LR
  SRC["Product DB / events"] --> RAW["Raw landing short TTL"]
  RAW --> VAULT["Identity vault restricted"]
  RAW --> COMPACT["Compacted progress minimized"]
  COMPACT --> MART["completion mart no direct IDs"]
  MART --> BOARD["Monday board aggregates"]
  VAULT -.->|controlled join| SUPPORT["Support tools"]
  POL["Classification retention purpose"] --> RAW
  POL --> MART
  POL --> VAULT

Deletion and access requests follow the catalog, not memory.

mermaidCopy
flowchart TB
  REQ["Verified privacy request"] --> CAT["Catalog and lineage"]
  CAT --> JOBS["Export or delete jobs per store"]
  JOBS --> RAW2["Raw / snapshots policy"]
  JOBS --> MART2["Mart rows"]
  JOBS --> EXT["Extracts and exports"]
  JOBS --> EVID["Evidence log"]

Technical Explanation

Where PII enters TechLearningPro's path

Progress events often carry learner_id and sometimes embed email when producers are sloppy. Contracts must reject direct identifiers on the analytics topic or strip them at the edge. Support and identity systems hold email in a vault database. The lake raw zone might temporarily hold wider payloads if the product bus does; TTL must be short and access tight. The completion mart should store learner_id, lesson_id, timestamps at the agreed grain, and definition_version—not email.

CI forbidden-column checks (lesson 59) and schema monitors (lesson 60) enforce the mart side. Contract tests at produce time enforce the topic side.

Row-level and column-level controls in practice

Enterprise tenants may require that company A's analysts never see company B's learner attempts. That is tenant isolation: row-level policies or separate physical schemas keyed by tenant, driven by lesson 57 patterns. Column-level masks hide email in views for roles that only need to troubleshoot completion counts. Do not implement tenant isolation only in the BI tool; the warehouse view or policy is the control. BI-only filters fail when someone connects with SQL.

Retention jobs and time travel

Pseudocode policy:

1. Raw objects: expire after 14 days via lifecycle rules; abort if landing still needs longer replay—then revisit Kafka retention and landing together (DataOps).
2. Mart detailed rows: retain 24 months; then roll to monthly aggregates without learner_id.
3. Snapshots: expire time travel beyond 7 days for tables with personal data unless a legal hold exists.
4. Extracts: regenerate from mart; delete files older than 7 days.
5. Backups: document lag until purge; do not claim instantaneous global erasure.

Legal holds freeze deletion for specific keys or periods. The job must check a hold table before purge.

Fulfilling deletion

textCopy
Input: verified learner_id, request_id
For each store in catalog.stores_for(personal_data=true):
  run store.delete_or_anonymize(learner_id)
  write evidence(request_id, store, rows_affected, status)
Reconcile: counts for learner_id == 0 in online stores (except hold)
Notify privacy ops with evidence pack

Anonymize when deletion would break non-personal aggregates that must remain: replace learner_id with a random nonce and drop joinability to the vault. Agree with counsel which approach applies.

Access and portability export

Export only from stores classified as holding that subject's data. Package as machine-readable JSON or CSV with a manifest. Do not include other learners' rows. Do not include secrets. Rate-limit and audit the export job. Prefer product-mediated download over emailing attachments.

Observability and CI without exfiltration

Lesson 60 already preferred counts over sample rows. Codify: monitor failure payloads may include attempt_id and lesson_id, not email. CI fixtures use synthetic learners. Staging is masked. DLP scanning can help shared drives; it does not replace minimization.

Records of processing as engineering inventory

Many privacy programs maintain a processing inventory. Your catalog of certified assets with purpose, ownership, and classification is the living input. Keep it near the code (ownership/*.yaml plus contract classification) so it cannot drift as easily as a yearly spreadsheet.

Children's data and age gates (engineering caution)

If TechLearningPro ever serves minors, retention, consent, and parental controls become stricter and jurisdiction-specific. Engineers should not invent under-13 pipelines casually. Flag age-related fields as sensitive, isolate them, and involve counsel before building analytics that profile children. This course's default teaching story assumes adult professional learners; if that assumption changes, classification and purpose must change first—not after a mart already exists.

Cross-border transfer as a pipeline concern

Copying progress raw to another region "for analytics latency" is a transfer of personal data. Lesson 58 priced the copy; this lesson asks whether the transfer is allowed and documented. Prefer regional processing for that tenant's data, or documented transfer mechanisms your counsel approves. Dual-region hot analytics of personal data without a privacy review is a compliance defect dressed as architecture.

Vendor subprocessors and the lake

When a SaaS observability or enrichment tool receives samples or syncs tables, that tool is a subprocessor for personal data if those tables include personal data. Minimize what you sync; prefer aggregates; list the vendor in the privacy program; kill syncs that include vault fields. Lesson 60's vendor-neutral monitors can run inside your account for that reason.

Quality versus privacy tension

Deduplicating learners sometimes tempts engineers to pull email as a merge key. Resist. Use learner_id from the system of record. If identity resolution is required, do it inside the vaulted identity service and emit only the resolved learner_id to analytics. Privacy and grain quality are not enemies if the join stays in the right system.

Encryption is necessary and not sufficient

Lesson 57 encrypts data at rest. Encryption does not classify columns, delete them on request, or stop a lawful-but-overbroad analyst role from reading email. Celebrating KMS as "GDPR done" fails interviews and audits. Pair encryption with minimization, access reviews, and retention jobs.

Differential access for support versus analytics

Support may need to look up a learner by email to fix a broken enrollment. Analytics needs attempt counts. Separate roles: support tools query the vault with heavy audit; analytics roles query the mart without email. Shared "data team admin" roles that can do both recreate the shared-drive incident with better SQL.

Contractual retention with enterprise customers

Enterprise contracts may demand longer attestation retention than your default consumer TTL. Encode tenant-specific retention in metadata (tenant_id → retention_days) and make purge jobs honor the maximum of policy and contract, or isolate enterprise attestation tables. A single global DELETE that breaks a customer contract is an avoidable outage of trust.

Practical Implementation or Design Walkthrough

Problem

Create controls for TechLearningPro learner and enterprise customer data on the progress → completion path, plus a sketch for a financial-adjacent export some enterprise customers demand (completion attestation). Include classification, mart minimization, masking for staging, retention, access review, and a deletion job outline. Complement lesson 57; do not rebuild cloud IAM from scratch.

Column classification snippet

yamlCopy
# contracts/completion_attempt.yaml
table: published.completion_attempt
purpose: "Certified completion analytics for learner progress boards"
retention_days_detailed: 730
columns:
  - name: attempt_id
    type: string
    classification: internal_identifier
    required: true
  - name: learner_id
    type: string
    classification: internal_identifier
    required: true
  - name: lesson_id
    type: string
    classification: operational
    required: true
  - name: event_date
    type: date
    classification: operational
    required: true
  - name: definition_version
    type: string
    classification: operational
    required: true
  - name: email
    type: string
    classification: direct_identifier
    required: false
    allowed: false  # CI fails if present

CI gate for forbidden classifications

pythonCopy
# tests/completion/test_no_direct_identifiers.py
import yaml

def test_mart_contract_forbids_direct_identifiers():
    contract = yaml.safe_load(open("contracts/completion_attempt.yaml"))
    for col in contract["columns"]:
        if col.get("classification") == "direct_identifier":
            assert col.get("allowed") is False
            assert col["name"] not in {"attempt_id", "learner_id", "lesson_id"}

Add a runtime schema test that production columns do not include email.

Staging mask

sqlCopy
-- Create masked staging view for developers
CREATE OR REPLACE VIEW staging.completion_attempt_masked AS
SELECT
  attempt_id,
  -- preserve cardinality, break real-world identity
  md5(cast(learner_id as varchar) || :staging_pepper) AS learner_id,
  lesson_id,
  event_date,
  definition_version
FROM staging.completion_attempt_raw;

Developers get the view, not the raw staging table. The pepper lives in a secret manager; not in git.

Retention lifecycle notes

textCopy
s3://tlp-progress-raw/  -> lifecycle expire 14 days
table published.completion_attempt -> job monthly_purge_detailed
  DELETE WHERE event_date < current_date - 730
table published.completion_monthly_agg -> keep longer without learner_id
ops.privacy_legal_holds -> checked before DELETE

Deletion job sketch

pythonCopy
# privacy/delete_learner.py
def delete_learner(conn, learner_id: str, request_id: str) -> None:
    holds = conn.execute(
        "SELECT 1 FROM ops.privacy_legal_holds WHERE learner_id = %s AND active",
        (learner_id,),
    ).fetchone()
    if holds:
        raise RuntimeError("legal hold active; refuse automated delete")

    for sql in [
        "DELETE FROM published.completion_attempt WHERE learner_id = %s",
        "DELETE FROM lake_meta.progress_index WHERE learner_id = %s",
        # raw object deletion may be prefix-based and async; record task id
    ]:
        result = conn.execute(sql, (learner_id,))
        conn.execute(
            """
            INSERT INTO ops.privacy_evidence
              (request_id, store_sql, rows_affected, completed_at)
            VALUES (%s, %s, %s, CURRENT_TIMESTAMP)
            """,
            (request_id, sql, result.rowcount),
        )

Raw object deletion often needs an inventory by learner_id written at ingest time; without an index, you cannot cheaply find files. Design the index early or accept that raw TTL is your primary raw-zone control.

Access review extract

sqlCopy
SELECT
  grantee,
  table_name,
  privilege,
  classification_max
FROM catalog.access_listing
WHERE classification_max IN ('direct_identifier', 'internal_identifier', 'sensitive')
ORDER BY table_name, grantee;

Owners mark keep/revoke. Automation opens revoke PRs or IAM changes through the same change control as lesson 59.

Attestation export for enterprise customers

Some customers need completion evidence without a full PII dump. Publish a purpose-built attestation table: tenant_id, opaque learner_token (tenant-scoped), lesson_id, completed_at, definition_version. Join to clear identities only inside the customer's IdP-controlled tool, not in a CSV on email. Classification: internal identifiers; retention per contract; access limited to that tenant's role via RLS from lesson 57 patterns.

Catalog entry template

yamlCopy
# catalog/published_completion_attempt.yaml
asset: published.completion_attempt
owners:
  product: analytics-board
  engineering: learning-analytics-de
  privacy_contact: privacy-ops
purpose: "Learning completion analytics; Monday board and tenant attestation source"
personal_data: true
direct_identifiers: false
columns_ref: contracts/completion_attempt.yaml
retention:
  detailed_rows_days: 730
  snapshot_days: 7
  raw_landing_days: 14
downstream:
  - board_sql
  - bi_extract_completion
  - attestation_export
upstream:
  - compacted_progress
  - completion_publish_dag
access_review_cadence: quarterly

Portability export sketch

pythonCopy
# privacy/export_learner.py
import json


def export_learner(conn, learner_id: str, request_id: str, path: str) -> None:
    rows = conn.execute(
        """
        SELECT attempt_id, lesson_id, event_date, definition_version, publish_time
        FROM published.completion_attempt
        WHERE learner_id = %s
        ORDER BY event_date, attempt_id
        """,
        (learner_id,),
    ).fetchall()
    # Intentionally no email, IP, or free-text fields
    payload = {
        "request_id": request_id,
        "learner_id": learner_id,
        "stores": ["published.completion_attempt"],
        "attempts": [dict(r) for r in rows],
    }
    with open(path, "w", encoding="utf-8") as f:
        json.dump(payload, f, default=str)
    conn.execute(
        """
        INSERT INTO ops.privacy_evidence (request_id, store_sql, rows_affected, completed_at)
        VALUES (%s, %s, %s, CURRENT_TIMESTAMP)
        """,
        (request_id, "export published.completion_attempt", len(rows)),
    )

Detect email-shaped columns in staging

sqlCopy
-- Heuristic guard; dialect varies. Ticket on hits.
SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'staging'
  AND table_name = 'completion_attempt_raw'
  AND (
    lower(column_name) LIKE '%email%'
    OR lower(column_name) LIKE '%phone%'
  );

Pair with a data probe that samples values for @ only in approved privacy tooling, never in CI logs.

Quarterly review checklist

1. Pull access listing for classified assets.
2. Owners keep/revoke with purpose notes.
3. Apply revokes through change control.
4. Confirm purge jobs succeeded each month (metrics).
5. Confirm no certified mart gained forbidden columns (schema monitor history).
6. Sample one deletion request evidence pack for completeness.
7. Record review date and participants in ops.privacy_reviews.

Enterprise Real-World Scenarios

TechLearningPro shared-drive export

An engineer exports one day of raw progress including emails to debug duplicates. Freshness monitors stay green. A privacy review finds the file. Response: delete the file, review access logs, revoke the broad raw read role that made the export easy, add forbidden-column and purpose checks so the certified path never needed raw for this debug, provide a restricted debug view with learner_id only and audited access. Post-incident: teach lesson 60 sample-row policy as a hard rule.

Northline financial close data

Northline mixes learner-like users with payment records for certifications. Payment tokens never belong in the analytics lake. A well-meaning ELT job flattens billing into the same mart "for convenience." Classification CI fails only if it exists—in this scenario it did not yet. Incident: remove columns, scrub snapshots per counsel, split billing into a vaulted financial store with separate owners and shorter access lists, keep completion mart minimized. Access reviews catch a contractor still reading the financial schema a quarter later.

GDPR-style deletion versus time travel

A learner requests deletion. The warehouse DELETE succeeds. Time travel still reads old snapshots for five days. The privacy pack claims "deleted" while an analyst with time-travel privilege can recover rows. Repair: expire snapshots on a privacy-compatible schedule; restrict time-travel privileges; document backup lag; never certify instantaneous erasure for offline backups without counsel sign-off. Observability should not page on expected snapshot expiry.

CCPA-like opt-out ignored by a marketing shadow mart

Product stores sale_opt_out. A shadow marketing table built from progress events ignores the flag. DataOps already wanted shadow marts gone; privacy adds urgency. Control: certified marketing outputs must join preference flags; CI tests a fixture user with opt-out; access to raw progress for marketing purposes is denied.

Best Practices

• Classify columns in contracts and catalogs; enforce with CI and schema monitors.
• Minimize certified marts; keep direct identifiers in vaulted systems.
• State purpose on assets; reject purposeless wide access.
• Implement retention as jobs across files, tables, snapshots, and extracts.
• Mask or synthesize non-prod; never clone full production PII by default.
• Review access regularly for humans and services; record evidence.
• Route privacy requests through catalog-driven jobs, not ad-hoc SQL heroes.
• Keep observability and tickets free of cleartext PII samples.
• Use lesson 57 RLS/IAM as mechanisms; use this lesson to decide what deserves them.
• Coordinate legal holds with purge jobs.
• Treat attestation and financial data as separate products with tighter owners.
• Measure governance by coverage of certified assets classified and review completion rates—not by tool logos.

Common Mistakes and Anti-Patterns

Putting email in the completion mart for "convenient joins." Joins belong to vaulted tools with purpose. The board does not need email.

Deletion only in the warehouse table. Snapshots, raw files, and extracts still hold data. Map the graph.

Calling hashed email anonymous. It may still be personal data; protect it accordingly.

Full production clones for staging. Lesson 59 and privacy both forbid the lazy clone.

BI-only tenant filters. SQL users bypass them. Enforce in the warehouse/lake policies.

Access reviews that skip service accounts. Notebooks and BI robots over-collect quietly.

Privacy evidence as screenshots without row counts. Store machine evidence from jobs.

DLP instead of minimization. Scanning shared drives is backup; stop exports at the source roles.

Ignoring free-text fields. Feedback holds unexpected PII; classify as sensitive and isolate.

Promising instant global erasure including cold backups without counsel. Document real timelines.

Duplicating lesson 57's entire IAM tutorial instead of adding classification and retention. Keep the complement sharp.

Troubleshooting and Design Review

Audit asks "where is email stored?" and engineers disagree. Catalog incomplete. Inventory certified and raw paths; add classification; make the vault the only cleartext email store.

Deletion job succeeds but support still sees the user. System of record is not the lake; coordinate product DB deletion and lake purge order. Catalog must include product stores or hand those to product engineering explicitly.

Staging unmasked after a refresh. ETL from prod overwritten the mask. Mask in views or transform on load; test for email-shaped columns in staging.

Access review lists thousands of users. Entitlements too coarse. Move to group-based access tied to purposes; fix IAM design with lesson 57 patterns.

Monitors alert with payloads containing PII. Fix notification templates; treat as a privacy bug.

Enterprise customer demands proof of retention. Show job metrics, lifecycle configs, and sample evidence logs—not a policy PDF alone.

Design review:

• Which direct identifiers exist outside the vault, and why?
• What is the retention job for each store on the certified chain?
• Can a developer query cleartext email in staging?
• When was the last access review for the vault and the mart?
• Does CI fail if email appears on the mart?
• How does a deletion request produce evidence?
• What does lesson 57 encrypt and audit that this lesson classifies and minimizes?

Interview Questions and Answers

Q1. How does this lesson differ from platform security?

Security focuses on authentication, authorization, encryption, and audit mechanisms. Privacy/governance engineering focuses on classification, minimization, purpose, retention, deletion, and proof. I use security controls to enforce privacy decisions; I do not treat IAM alone as a privacy program.

Q2. Why keep learner_id but drop email from the completion mart?

learner_id supports grain and per-learner analytics without exposing direct contact identifiers. Email belongs in an identity vault with tighter access. The board aggregates do not need email. Minimization reduces breach impact and access-review burden.

Q3. What stores must a deletion request consider?

Online marts, raw landing, indices that map learners to files, table snapshots/time travel, BI extracts, downstream replicas, and documented backup timelines. Catalog and lineage drive the checklist; memory does not.

Q4. How do you make staging safe?

Masked or synthetic data with schema parity; no default full prod clones; CI tests that reject email-shaped columns; separate secrets and identities from prod.

Q5. What is purpose limitation in pipeline terms?

Only process columns and populations needed for a stated purpose; new purposes need new access grants and often new models. A completion mart should not quietly become a marketing CDP.

Q6. How do GDPR-style access requests become engineering work?

Verified identity upstream; catalog identifies stores; export job pulls only that subject's rows; audit the export; package with a manifest. Avoid one-off SQL that misses systems or includes others' data.

Q7. Why are free-text feedback fields dangerous?

Users paste emails, names, and sensitive stories into free text. Classify as sensitive, isolate storage, shorten retention, and restrict access even if the parent event stream looks operational.

Q8. How do access reviews connect to DataOps ownership?

Owners from lesson 59 attest who should still read their assets. Reviews produce revoke actions through change control. Unowned tables cannot be reviewed coherently—fix ownership first.

Q9. Can you rely on hashing email for anonymization?

Usually not as a blanket claim. Hashes can be reversible with rainbow tables if unsalted/unpeppered, and may still be personal data. Prefer vaulting, minimization, and access control; if hashing, use pepper and counsel guidance.

Q10. How should observability avoid privacy regressions?

Alert on aggregates; deep-link runbooks; put sample-row debugging in audited tools; never Slack cleartext PII; classify monitor payloads.

Q11. What evidence satisfies a retention audit better than a policy PDF?

Lifecycle configurations, scheduled purge job success metrics, sample evidence logs for deletes, and snapshot expiry settings—plus owners who can explain exceptions and legal holds.

Q12. How would you handle an enterprise attestation export?

Purpose-built minimized table with tenant RLS, opaque tokens instead of emails, pinned definition_version, contract retention, and no hitchhiking payment fields. Clear identities resolve in the customer's controlled environment when required.

Hands-On Exercises

Exercise: Controls for learner and customer data

Problem statement. Create governance and privacy controls for TechLearningPro's progress and completion data, including an enterprise attestation-style export path. Cover classification, minimization, staging masking, retention across stores, access review, and deletion evidence. Complement—not redo—lesson 57 IAM/KMS depth. Assume owners from lesson 59 and monitors from lesson 60 exist.

Tasks.

1. Classify columns for raw progress landing, completion mart, and attestation export.
2. Write contract/CI rules that fail if direct identifiers appear on the mart.
3. Design staging masking and state what prod debug is allowed.
4. Map retention actions for raw files, mart rows, snapshots, and extracts.
5. Outline a deletion job with legal-hold check and evidence logging.
6. Define a quarterly access-review query and owner workflow.
7. Explain how a privacy request uses the catalog/lineage graph.
8. List non-goals (what counsel decides; what lesson 57 already covered).

Expected deliverables. Classification table or YAML, CI rule description, retention map, deletion outline, access-review plan, attestation export sketch, non-goals.

Evaluation rubric.

Privacy incident runbook (engineering page)

1. Contain: revoke shared links, rotate exposed credentials, restrict overbroad roles.
2. Inventory: what classes of data, which stores, approximate volume, time range.
3. Preserve evidence: access logs from lesson 57, export paths, ticket threads—without spreading PII further on Slack.
4. Notify privacy ops / counsel per policy timelines; engineers do not freelance external notification.
5. Eradicate: delete or quarantine copies; fix the control gap (role, CI, mask).
6. Post-incident: add CI or monitor; schedule access review early; update catalog.

How Module 6 lessons fit together

Lesson 59 decides who may change the platform. Lesson 60 decides how you notice data failure. Lesson 61 decides what personal data may exist and how it disappears. Lesson 62 will decide how late is too late. None substitutes for another. A fresh, owned mart that holds email for convenience still fails this lesson.

Capstone preview

The enterprise capstone will ask you to ship a path that is operable. Expect to show classification on the mart, a deletion or retention job, and an access boundary between vault and analytics—alongside freshness and cost. Build those muscles here so the capstone is assembly, not invention under deadline.

Common counsel questions engineers should anticipate

• Where do you store email cleartext today?
• How long do raw logs live?
• How do you fulfill deletion across backups?
• Who approved the marketing sync?
• Can a contractor still read production?

If your catalog and jobs can answer, you are engineering compliance. If only slide decks can answer, you are hoping.

Worked minimization for the progress event

Suppose the product bus emits:

learner_id, email, lesson_id, event_time, client_ip, user_agent, raw_payload, definition_hint.

Analytics purpose for the completion board needs: learner_id, lesson_id, event_time (or event_date), and the fields required to apply completion-v4. Drop email at ingest to analytics topics, or strip in the first controlled transform. Store client_ip only if fraud analytics has a purpose and short TTL; do not copy it into the completion mart. Keep raw_payload in a short-TTL raw zone if replay needs it; do not project it into Parquet the board reads. user_agent may be operational; still consider it a quasi-identifier when combined with tenant and time. The mart publish SQL is an allow-list of columns, never SELECT * from raw—the same rule lesson 58 and 59 already wanted for cost and CI.

Idempotent privacy jobs

Deletion and export jobs must be safe to rerun. Evidence rows use request_id uniqueness. Deletes that find zero rows after a prior success record already_clean rather than failing the privacy SLA. Exports overwrite the same sealed object path or write versioned objects listed in evidence. Partial failure leaves a retryable state machine: which stores completed, which remain. Privacy ops should see a checklist, not a single boolean that lies.

Logging discipline

Application logs that print the progress event body will recreate email in CloudWatch/Stackdriver/etc. with longer retention than the lake. Ban payload logging on certified paths; log ids and hashes. Log redaction libraries help; minimization helps more. Lesson 60 alerts that include the failing SQL with literals can leak identifiers—use bind parameter placeholders in notification text.

Training data and model features

If a future ML feature store copies learner sequences, it becomes another personal-data store with its own retention and deletion path. Do not discover it during a deletion request. Register it in the catalog when it is born. Prefer learner_id features over cleartext profile fields. Governance debt in feature stores is still governance debt.

When counsel and engineering disagree on timelines

Counsel may require a thirty-day deletion SLA for online systems and accept longer for cold backups. Engineering must encode the online SLA in jobs and monitoring (ticket if a request stays open past SLA) and document backup realities in the evidence pack template. Do not silently promise backup erasure you cannot perform. Escalation is a written exception, not a hopeful chat message.

Northline payment fields—control detail

Payment primary account numbers or tokens never land in the analytics lake. Billing events that must support "paid certification" emit payment_status and entitlement_id only. Enrichment joins to billing vault happen in a restricted service, not in a shared Spark job with broad output. CI forbidden lists include card, pan, cvv, account_number column names. Schema monitors ticket on matches. This is governance as boring string denylists plus architecture—both required.

Measuring governance coverage

Track: percent of certified assets with classification complete; percent with retention job linked; percent with privacy contact; access review completion on time; deletion request SLA hit rate; count of direct identifiers found in non-vault certified tables (target zero). Publish these next to DataOps ownership coverage. Tools do not equal coverage.

Joining vault data safely for rare investigations

Occasionally analytics must answer "which email failed enrollment" during an incident. Do not widen the mart. Spin a time-bounded investigation workspace: vault join allowed for named people, query results expire in twenty-four hours, downloads disabled, session audited. The investigation SQL is reviewed like a high-risk change. When the incident ends, revoke the workspace. This pattern beats permanent email columns "just in case."

Aggregates that still identify people

A board tile filtered to one enterprise tenant with three learners can re-identify individuals even without names. Apply minimum cohort thresholds for external or widely shared dashboards (for example, suppress cells with n under ten). Internal ops tiles with tight IAM may show finer grain. Classification of the output matters, not only of the input columns. Lesson 60's distribution monitors should not publish small-n breakdowns to public channels.

Consent and preference flags in ELT

If marketing_ok or analytics_ok flags exist in the product database, ELT that builds optional marketing marts must filter them. Add CI fixtures: a learner with marketing_ok=false must not appear in marketing outputs. Completion board certified paths typically process operational learning analytics under a different purpose—document that purpose so flags are not misapplied or ignored. Preference flags are data contracts, not decorations.

Backup and disaster-recovery personal data

DR replicas contain personal data by definition if the primary does. Protect them with equal IAM severity, encrypt them, and include them in the deletion policy narrative (erase on cycle versus online delete). A "cold" replica that every engineer can read is not cold. Lesson 58's restore drill should use masked or synthetic data when possible; if it must use production, treat the drill environment as production-classified for the drill window.

Why catalogs fail in enterprises—and how to avoid it

Catalogs fail when updates are a separate human chore after every deploy. Generate catalog stubs from contract YAML in CI; fail the build if ownership or classification keys are missing for certified assets. Humans still write purpose prose; machines catch empty owners. That is the same DataOps instinct applied to governance metadata.

Hands-on note on financial-adjacent fields

The brief asked for controls for customer and financial data. For TechLearningPro, "financial" usually means certification purchase entitlements, not card PANs. Entitlement status can live in analytics; payment instruments cannot. If your org truly needs finance marts, give them a separate domain owner, separate warehouse database, separate access review, and forbid joins into the learning completion mart except through non-sensitive entitlement ids. Mixing finance and learning in one wide table is how over-broad access becomes inevitable.

Retention calendar example for the teaching volumes

At about 40 GB/day of compacted Parquet, fourteen days of raw (if still kept) and two years of detailed mart rows are manageable when lifecycle jobs run. The surprise growth from lesson 58 often came from unbounded raw and replicas—the same stores privacy wants bounded. Align the cost sheet's lifecycle lines with the privacy retention map so finance and counsel are not given different stories. When product extends attestation retention for an enterprise tenant, record the exception in metadata and in the cost forecast.

Interview follow-through

In interviews, candidates who only say "we encrypt everything" fail this lesson's bar. Strong answers name classification, minimization of the mart, a deletion graph, access reviews, and the separation from IAM mechanics. Tie answers to TechLearningPro: the board does not need email; the vault does; staging is masked; snapshots expire; evidence is logged.

Lesson Summary

Governance and privacy turn personal data from an accident into a designed constraint. Classify columns. Minimize the mart. Mask non-prod. Retain and delete across the whole graph. Review access. Express regulatory patterns as jobs and catalog fields. Use lesson 57's identities and audits as the mechanism layer; do not stop at mechanism. The Monday board can be fresh, cheap, and owned—and still fail if it becomes a convenient place to keep email.

Data Reliability and SLA/SLO Design will bind operational promises; keep privacy promises equally concrete. The capstone must pick engines and pipelines that still know what they hold about learners and how to stop holding it when required.

Further Reading

• ICO GDPR guidance and EU GDPR portal for obligation themes—pair with your counsel; this lesson is engineering translation, not legal advice.
• California CCPA/CPRA resources for consumer rights patterns that affect US-facing products.
• NIST Privacy Framework for organizational privacy engineering structure.
• Cloud DLP / sensitive-data discovery docs for your cloud (AWS, Azure, GCP) as assistive tooling—not a substitute for minimization.
• Apache Iceberg docs / Delta Lake docs for snapshot expiry as a retention control.
• Lessons 22, 57, 59, and 60 for catalog, security mechanisms, ownership, and PII-safe monitoring.
• Your organization's records-of-processing and retention schedule templates; align catalog fields to them.