Data Engineering Roles and Responsibilities

Data Engineering & Modern Data Platforms · Lesson 03 · Data Engineering Foundations

Lesson Overview

The completion mart from lessons 1 and 2 was green on a Wednesday, and the Monday number was still wrong. A product manager asked who would fix it. The analyst who wrote the original definition was on leave. The data engineer who owned the job said the SQL matched the ticket. The platform engineer said the cluster was healthy. Product engineering said the player had not changed. Four people were present. None of them was accountable for the meaning of the row, and none of them was allowed to stop the other dashboard that still read definition v2.

That is a role failure, not a SQL failure. Data engineering work sits on a boundary between people who generate facts, people who decide what a metric means, people who run shared platforms, and people who get paged. Job titles vary by company and are a poor guide. The useful map is a responsibility assignment: for each dataset, pipeline, and incident class, who is accountable, who is responsible for the change, who must be consulted, and who is informed. RACI is an old project-management device. Applied to a data product it is an ownership contract the on-call rotation can execute.

This lesson builds that contract for the TechLearningPro analytics estate. It does not hire a fictional department. The planning assumption is still a small company: one data engineer, one analytics engineer, analysts who sit with product, a product engineering squad that owns the player and enrollment, and a single platform person who also carries other systems. The map has to fit those people. A RACI that names six teams you do not have is the reference-architecture problem from lesson 2, drawn as an org chart.

You will leave able to assign the completion mart, the raw extract, and a broken metric to named accountability, to choose a team shape without pretending Conway's law is optional, and to reject a design whose pager does not land on a person. Lesson 59 expands DataOps. Lesson 62 expands SLAs as numerical objectives. This lesson sets the owner those later numbers attach to.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate accountability for a data product from the several roles that contribute to it.
2. Place data engineers, analytics engineers, analysts, scientists, platform engineers, and product engineers on one learning-platform incident.
3. Write a RACI row for a pipeline, a metric definition, a shared platform, and an access grant.
4. Choose among centralized, embedded, and hub-and-spoke team shapes using staffing and Conway's law, not fashion.
5. Define an on-call duty that a small squad can actually carry, including what they will not be paged for.
6. Explain why "the warehouse team owns data quality" is too coarse to be an incident process.
7. Encode a minimal ownership check in Python and find an unowned dataset with SQL.
8. Review a responsibility map and reject a row that has two people accountable, or none.

Prerequisites

• Lessons 1 and 2: the completion mart, its grain, its publication row, and the shapes (operational database, raw zone, curated table, serving API). This lesson assigns owners to those shapes. It does not redraw them.
• You can read a short Python dataclass and a SQL SELECT. Language depth is lessons 4 through 6.
• You have worked on a team that shipped software someone else operated, or operated software someone else shipped. The friction between those states is the lesson.

Organizational words are defined when they appear. You do not need a project-management certificate. You do need to accept that "we all own it" means nobody is paged.

Detailed Core Concepts

Accountability is singular

Responsible means the person who does the work. Several people can be responsible for parts of a change.

Accountable means the person who answers for the outcome and who can say no. Exactly one accountable role per decision. Two accountable people produce the Wednesday incident: each can claim the other approved the definition. Zero accountable people produces the same incident with more meetings.

Consulted means the work is not done until this person has been heard. Consultation is not a veto unless the RACI says so. Product engineering is consulted on any change to the event contract, and they do hold a veto on the wire format, because they own generation. That veto is a consulted-with-veto cell, written explicitly. A vague "consult product" cell is how a backfill ships a new meaning the player cannot emit.

Informed means the person is told, not asked. The Monday review attendees are informed when definition v3 is restated. They do not edit the job.

RACI gets abused when every cell is filled to look thorough. A row with four A's is a row that has not been decided. A row with an A and no R means the accountable person is expected to perform work they cannot see. Keep the matrix small enough to use during an incident: one page for the completion product, not a catalog of the company.

Roles you will actually meet

Titles drift. The responsibilities below are the ones this course means when it uses the title. A person may wear two of them in a small company. The RACI still names the hat they are wearing for that decision, because the conflict of interest is real: the person who wants the dashboard today is a poor sole approver of the definition they will be rewarded for moving.

Analytics engineer, in the sense popularized by teams that treat transforms as software, sits between the analyst and the data engineer. In a large company the hat is a job. In TechLearningPro's current assumption it is a hat the data engineer and the senior analyst share, with the analyst accountable for the wording and the data engineer accountable for the job that implements the wording. Writing both names in the A column is forbidden. The matrix in the walkthrough picks one A per row and puts the other in C or R.

Data architects who only draw and never see an incident lose the right to the title. The useful architect writes the reopening condition and attends the review when it trips. Platform engineers who own every pipeline become a queue. Domain data engineers who each build a private orchestrator become an estate nobody can patch. The team-shape section is that tension.

What "ownership of the data" must specify

"Who owns the data?" is four questions pretending to be one.

1. Source ownership. Who may change the generating schema? Product engineering, for progress events.
2. Definition ownership. Who may change what completion means? One accountable person, with analytics responsible for the draft and the data engineer responsible for the replay plan.
3. Pipeline ownership. Who is paged when the publication does not land by 06:00 UTC? The data engineer on the rotation.
4. Access ownership. Who may grant a new human or service the right to read the mart? The platform engineer, against a purpose recorded by governance. The data engineer does not hand out credentials because someone asked in chat.

A team that answers all four with "data" will grant the warehouse password to the person who asked the question, change the metric in the BI tool, and leave the player contract untouched. The incident looks like a quality problem. It is an ownership split that was never written.

Quality itself splits the same way. Completeness of the extract is a pipeline duty. Validity of event_id on the wire is a product-engineering duty. Semantic correctness of completion-v3 is a definition duty. Referential fit to the course catalog is a joint duty: product owns catalog identity, the transform owns the quarantine when an id does not resolve. A single "data quality owner" can sponsor the program. They cannot be the only name on every defect class. Lesson 17 builds the tests. The routing of those tests is decided here.

Team shapes

Three shapes cover almost every organization you will be asked to defend.

Centralized. One data team takes tickets from the company. It is simple to staff and easy to bottleneck. Domain context lives in the ticket, which means it is already gone. This shape fits a company with one or two data people and a handful of sources, which is the TechLearningPro assumption today. It stops fitting when the ticket queue is the product, and analysts route around it with extracts.

Embedded. A data engineer sits inside each product squad. Context is high. Platform consistency is low unless something else pulls the paved road. Embedded teams duplicate ingestion patterns and prohibited-field rules. Conway's law appears as a warehouse per squad that cannot be joined. Use embedding when the domain's event contract changes weekly and a central queue would always be late. Do not use it as a slogan about autonomy when the company has one data engineer. That person cannot be embedded in four squads except on a slide.

Hub and spoke. A platform hub owns engines, paved paths, identity, and standards. Spokes own domain products: the completion definition, the finance fact, the backfill of their grain. The hub does not take metric tickets. The spoke does not run a private Kafka cluster. This is the shape Northline Marketplace in lesson 2 will need once a second domain (payouts) arrives. It is heavier than TechLearningPro needs while there is one mart. The decision record should say "centralized now, hub and spoke when a second domain product has a different pager," which is a reopening condition, not a reorganization rumor.

Conway's law, as Conway wrote it, observes that systems copy the communication structure that produces them. A RACI that pretends four teams jointly own one table will produce four writers. A RACI that names one accountable owner per product produces one publication path. Org design is an architecture constraint. Lesson 2 already used that idea for engines. Here it applies to people.

On-call that a small squad can survive

An on-call duty is a list of pages and a list of non-pages.

Pages for the TechLearningPro data rotation:

• mart_publication for yesterday's event day is missing at 06:15 UTC.
• The contract checker rejected a batch, or quarantine exceeded the agreed count.
• A prohibited field was detected in a landed object.
• The extract role cannot read the replica, so the raw zone will go stale.

Does not page the data rotation:

• The player is down. Product on-call owns that. Data is informed if the gap will miss the publication, and the user-visible status is not "the warehouse is wrong."
• An analyst disagrees with a number during the workday and the publication succeeded. That is a definition ticket, handled in business hours, unless the disagreement is a restatement that changed a closed financial figure. Finance figures are not yet on this mart. When they are, the page list changes in writing.
• A dashboard timed out because someone selected three years of raw JSON. That is a serving-path bug to fix later, and a permissions fix if they should not have scanned raw data. It is not a 02:00 rewrite of the metric.

The rotation has two humans in the assumption, so the "follow the sun" diagram some handbooks draw is a fiction. Severity is how you protect them. A missed publication is a page because consumers were promised 06:00. A chart color is not. Lesson 62 will turn "promised" into an error budget. Until then, the promise is the freshness sentence from lesson 1, and the owner is the rotation.

Service levels are someone's name plus a number

A service level without an owner is a slogan. For this lesson the minimum record is:

The consumer behavior column is the part teams skip. If the studio shows zero when the job is late, the instructor believes completions vanished. The serving contract is to show staleness. That is a product requirement the data owner is accountable for specifying and the studio engineer is responsible for implementing. Both names go in the RACI. Only one is accountable for the promise "we will not display a false zero."

Interfaces between roles, as artifacts

Conversations do not survive vacation. The interfaces that do:

• Event contract. Fields, types, idempotency, prohibited keys. Product engineering proposes a change; the data engineer updates the checker; both sign. Lesson 1's Python module is the start of this artifact.
• Metric spec. Grain, filters, clock, version, consumers. The analyst writes the draft. The accountable definition owner accepts it. The transform implements only an accepted version.
• Backfill plan. Which days will be replaced, what consumers will see, when. A backfill without a plan is how Tuesday changes during a board meeting.
• Access request. Purpose, dataset, duration. Platform grants against the request. Chat screenshots are not requests.
• Incident note. Which of the four ownership questions the defect belongs to, what was restored, what ticket changes the contract.

If a role has no artifact, the role is a meeting. Meetings are sometimes necessary. They are not a control.

Architecture and Design Diagrams

Ownership is a data flow of decisions, not of bytes. The bytes still move as in lesson 2. This diagram shows who can change each stage.

mermaidCopy
flowchart LR
    subgraph gen [Generation]
        pe[Product engineering: accountable]
        wire[Event contract]
        pe --> wire
    end

    subgraph pipe [Pipeline]
        de[Data engineer: accountable]
        raw[Raw batch and quarantine]
        job[Publication job]
        de --> raw --> job
    end

    subgraph meaning [Definition]
        an[Analytics: accountable for wording]
        spec[Metric spec vN]
        an --> spec
    end

    subgraph serve [Serving]
        api[Studio engineer: responsible]
        pub[Published snapshot]
        api --> pub
    end

    wire --> raw
    spec --> job
    job --> pub
    gov[Governance: consulted on fields and grants]
    gov -.-> wire
    gov -.-> raw
    plat[Platform: accountable for grants and the engine]
    plat -.-> job

The dotted arrows are consultation or platform control, not a second writer of the metric. Governance does not publish a completion row. Platform does not edit the spec. Product engineering does not hot-fix the mart in production to match a player change that has not been versioned.

A second picture is the incident path when the 06:00 publication is missing. It is the flow an on-call engineer should be able to walk without a channel argument.

mermaidCopy
flowchart TD
    page[Page: publication missing at 06:15 UTC]
    last{Last good publication still readable?}
    show[Serving keeps showing last publication and its timestamp]
    cause{Which ownership class?}
    source[Product defect: events stopped or contract broke]
    pipe[Pipeline defect: extract, quarantine, or job]
    define[Definition defect: spec and job disagree]
    page --> last
    last -->|yes| show
    last -->|no| pipe
    show --> cause
    cause -->|wire or player| source
    cause -->|job or files| pipe
    cause -->|meaning| define
    source --> pe[Product on-call accountable]
    pipe --> de[Data on-call restores or replays]
    define --> daytime[Ticket for business hours unless a closed figure moved]

The diagram's important branch is the serving behavior while the cause is unknown. Stale and labeled beats fresh and invented. Teams that skip that branch "fix" the incident by running an analyst query and pasting a number into the channel. The paste becomes the new unofficial product.

Hub and spoke, for the day a second domain arrives, is an allocation of people onto those same boxes.

mermaidCopy
flowchart TB
    hub[Platform hub: engines, identity, paved job runner]
    completion[Spoke: completion product]
    payouts[Spoke: payout fact, later]
    hub --> completion
    hub --> payouts
    completion -.->|no private orchestrator| hub
    payouts -.->|no private copy of learner email| hub

Until the payout spoke exists, the hub and the completion spoke are the same two people wearing different hats, and the RACI is what keeps the hats from blending into "whoever is online."

Technical Explanation

RACI as data, not as a slide

A responsibility matrix that lives in a presentation will be wrong within a month, because the presentation is not reviewed when the job changes. Store the rows next to the pipeline. The walkthrough uses a Python structure the repository can test, and a SQL table a reviewer can query. The two must agree. A grant that exists in the database and not in the matrix is an access incident. A matrix row with no pipeline is fiction.

The check that matters mechanically:

• Each decision_id has exactly one accountable row.
• The accountable person is a known role, not a distribution list.
• A pipeline row names a page_condition.
• A dataset marked served names a publication owner.
• No row assigns accountability to "team" or "everyone."

Judgment remains for the consulted list. The machine only rejects matrices an incident cannot execute.

Severity and response targets

Do not copy a generic Sev-1 table from an infrastructure handbook without editing the consumer column. For this estate:

A Sev-1 that includes "executive is unhappy" will train the rotation to bypass the spec. Unhappiness is not a severity. A wrong official number is.

Response here means a human starts looking, not that the definition is redesigned overnight. The only overnight change allowed to the mart is a revert to a prior publication or a replay of an accepted spec. New semantics wait for the accountable definition owner.

Conway effects you can predict

If analysts are the only people who can ship SQL to production, the warehouse will look like a folder of personal schemas. If product engineers cannot see the contract tests, the payload will gain fields the checker rejects on the worst day. If platform owns the transform repository, every metric change waits on a team that does not sit with the consumer, and spokes will start a second warehouse. The RACI is the counter-pressure. Put the definition change in the spoke's review, the engine upgrade in the hub's review, and the event contract in the product squad's review, with the data checker as a required check on that pull request.

Required checks are the technical form of "consulted." A pull request that changes attempt_number fails if the contract module's tests fail. A pull request that changes the metric SQL fails if the spec version was not bumped. Those checks are how a two-person team gets the behavior of the matrix on a week when nobody rereads it.

Handoffs that need a watermark, not a meeting

Product engineering hands events to data engineering at a watermark: the replica position or the batch id that is complete. Data engineering hands a publication id to serving. Serving hands a displayed timestamp to the instructor. Each handoff has a record. "We synced in Slack" is not a watermark. When the publication is late, the on-call engineer looks at the last successful batch id and the last publication row. If you cannot query those, the role map is theater, because the accountable person cannot see the boundary they own.

Staffing the map honestly

The assumption in numbers: one data engineer, one analyst who can review SQL, two product engineers on the player, one platform engineer shared with the product. The data engineer is accountable for the pipeline and is on call every other week with the platform engineer as backup, and the backup's duty is restore-to-last-publication, not metric redesign. The analyst is accountable for the wording of completion-v3 and is not on the night rotation. Writing the analyst into the night rotation because "they know the metric" guarantees a bad definition change at 02:00 and a resignation later.

If the company adds a payout fact that finance will close the books on, the map does not silently extend the same rotation. Finance closure is a new severity and a new spoke. The reopening condition for team shape is that second pager, or a queue of metric changes waiting more than two weeks. Until then, centralizing the work in the two data-capable people is the design. State it so nobody "helps" by creating a new schema in their personal project.

Failure of shared ownership, in one transaction

Two writers to lesson_completion are an ownership defect the database can express. The publication job uses a role that is the only role with insert and delete on that table. The analyst role can select published rows and cannot create tables in the mart schema. The raw bucket policy allows the extract role to put and the job role to get. An analyst credential that can also put objects will eventually "repair" a day by uploading a CSV. Revoke it. The RACI and the grants are the same decision seen by humans and by machines.

Practical Implementation or Design Walkthrough

Problem

Encode the completion product's RACI so a pull request or a nightly check can reject an unowned decision, and expose the same rows in SQL for a design review. The example uses the small-team assumption. Names are roles, not employees, so the file does not rot when someone leaves. A separate roster maps roles to people and is out of this lesson's code on purpose: the contract is about hats.

Decisions that need a row

monday_review and instructors are informed audiences, not roles that hold credentials. The checker allows them only in the informed column.

Python

Python 3.11+, standard library. One accountable entry per decision. Distribution-list names are rejected.

pythonCopy
from dataclasses import dataclass

ROLES = frozenset({
    "product_engineering",
    "data_engineering",
    "analytics",
    "platform",
    "governance",
    "product_management",
    "studio",
})
AUDIENCES = frozenset({"monday_review", "instructors"})
BANNED_OWNERS = frozenset({"team", "everyone", "data", "shared"})


@dataclass(frozen=True)
class RaciRow:
    decision_id: str
    accountable: str
    responsible: tuple[str, ...]
    consulted: tuple[str, ...]
    informed: tuple[str, ...]
    page_condition: str | None = None

    def __post_init__(self) -> None:
        if not self.decision_id.strip():
            raise ValueError("decision_id is required")
        if self.accountable in BANNED_OWNERS or self.accountable not in ROLES:
            raise ValueError(f"accountable must be a single role: {self.accountable}")


def review_matrix(rows: tuple[RaciRow, ...]) -> tuple[str, ...]:
    findings: list[str] = []
    seen: dict[str, str] = {}
    for row in rows:
        if row.decision_id in seen:
            findings.append(f"{row.decision_id}: more than one row")
        seen[row.decision_id] = row.accountable
        if not row.responsible:
            findings.append(f"{row.decision_id}: nobody is responsible")
        for name in row.responsible + row.consulted:
            if name not in ROLES:
                findings.append(f"{row.decision_id}: {name} is not a role")
        for name in row.informed:
            if name not in ROLES and name not in AUDIENCES:
                findings.append(f"{row.decision_id}: {name} is not informed-legal")
        if row.decision_id.startswith("pipeline.") and not row.page_condition:
            findings.append(f"{row.decision_id}: pipeline rows need a page condition")
    return tuple(findings)


def _demo() -> None:
    good = RaciRow(
        decision_id="pipeline.completion_publication",
        accountable="data_engineering",
        responsible=("data_engineering",),
        consulted=("analytics", "platform"),
        informed=("studio", "monday_review"),
        page_condition="no publication for yesterday at 06:15 UTC",
    )
    assert review_matrix((good,)) == ()
    try:
        RaciRow(
            decision_id="metric.wording",
            accountable="everyone",
            responsible=("analytics",),
            consulted=(),
            informed=(),
        )
    except ValueError as exc:
        assert "single role" in str(exc)
    else:
        raise AssertionError("everyone must not be accountable")
    print("raci_ok=1")


if __name__ == "__main__":
    _demo()

SQL the review can run

PostgreSQL 16. This table is the review copy of the same decisions. A nightly test can export the Python rows and compare them. Do not maintain the SQL by hand in a second meeting.

sqlCopy
CREATE TABLE ownership_matrix (
    decision_id text PRIMARY KEY,
    accountable text NOT NULL CHECK (accountable NOT IN ('team', 'everyone', 'data', 'shared')),
    page_condition text,
    CONSTRAINT pipeline_has_page CHECK (
        decision_id NOT LIKE 'pipeline.%' OR page_condition IS NOT NULL
    )
);

INSERT INTO ownership_matrix (decision_id, accountable, page_condition) VALUES
    ('pipeline.completion_publication', 'data_engineering',
     'no publication for yesterday at 06:15 UTC'),
    ('metric.completion_wording', 'analytics', NULL);

-- Decisions that cannot page anyone are visible, not hidden.
SELECT decision_id
FROM ownership_matrix
WHERE decision_id LIKE 'pipeline.%' AND page_condition IS NULL;

The SELECT is the design-review query. It must return zero rows. The check constraint makes the insert fail before the select matters. Both exist because a constraint protects the write and a query gives the reviewer something to paste into a ticket.

What the walkthrough does not do

It does not integrate with an HR system, a paging vendor, or a cloud IAM API. Those are real and specific to a company. Inventing a client for them would pretend TechLearningPro has a platform it has not specified. The artifact is the matrix and the grant discipline. Wire the page condition to whatever on-call tool you actually run, and keep the condition text identical to the row.

Enterprise Real-World Scenarios

The Wednesday wrong number

The publication had succeeded. Definition v3 was in the job. A personal dashboard still computed v2 from raw JSON using a credential an analyst had been given during the incident in lesson 1. The RACI that should have existed would have made three defects routable.

• Access ownership: platform was accountable for grants and had never recorded the personal credential. The repair is a revoke and an access-request artifact. The data engineer does not get to "just add them back" during the next incident.
• Definition ownership: analytics is accountable for wording. The personal dashboard was an unapproved spec. The repair is to delete it or point it at the published mart. Informed parties in the Monday review are told which version is official.
• Pipeline ownership: the data engineer was accountable for publication and had met the promise. Paging them to "fix the number" overnight would have produced a third definition. The incident commander, who is the data on-call only if the publication itself is wrong, routes meaning disputes to business hours.

After the repair, the studio shows definition_version and published_at in a footer an instructor can read. That footer is the studio row of the RACI, and it would have made the personal dashboard obviously unofficial. The technical change is small. The role change is that someone is allowed to revoke a credential without a debate about being a team player.

A responsibility map when the second domain arrives

Northline Marketplace, the staff-training case, hires a second data engineer for payouts. Leadership proposes embedding one engineer in "growth" and one in "finance," with no platform owner, because the word embedded sounds modern. The operational consequence is easy to predict. Growth builds a nightly script on a VM. Finance builds a different orchestrator. Both copy shopper email into their projects because the identity join is convenient and governance was not a role on either squad.

The map the case adopts instead:

• Hub: the original platform engineer, accountable for the job runner, warehouse roles, and the prohibited-field check every spoke must call. The hub is not accountable for payout math.
• Completion spoke: the existing data engineer, accountable for the completion publication, consulted on any event-contract change.
• Payout spoke: the new data engineer, accountable for the payout fact. Finance is consulted and holds a veto on the money grain. The spoke may not create a cloud account.
• Governance: one named person in legal or security, consulted on both spokes' projections, accountable for the prohibited-field list itself.
• Product engineering: still accountable for generation, with one contract per event type. Payout events are not stuffed into the progress payload "so there is only one stream."

On-call splits by publication. A missed completion page does not ring the payout engineer, and a payout restatement outside the refund window does not get "fixed" by the completion job. Shared infrastructure pages ring the hub. If the hub is a single person, the backup is restore-only, and the reopening condition for hiring is two Sev-2 pages in a month that the backup could not diagnose. That condition is a staffing plan. The diagram with six boxes and no names was not.

Best Practices

• One accountable role per decision. Put "everyone" in the informed column or nowhere.
• Split ownership into source, definition, pipeline, and access. Ask which of the four a defect belongs to before changing code.
• Store the RACI next to the pipeline and reject unowned pipeline rows in CI.
• Make grants match the matrix. The role that is not accountable for repair should not be able to write the mart.
• Page on missed publication, quarantine breaches, and prohibited fields. Do not page on metric disagreement when the publication matched the accepted spec.
• While the cause is unknown, serve the last publication with its timestamp. Do not paste a replacement number from a laptop.
• Choose centralized, embedded, or hub-and-spoke from the number of domains and pagers you have. Write the condition that changes the shape.
• Turn "consulted" into a required check on the pull request that changes the contract or the spec version.
• Keep night rotations from redesigning semantics. Overnight work replays or reverts.
• When one human wears two hats, the matrix still names the hat for the decision. The conflict is the reason the second name is consulted rather than jointly accountable.

Common Mistakes and Anti-Patterns

The data team owns data. A single blob of accountability absorbs product defects, metric politics, and access requests. The team becomes a queue, then a bottleneck, then a bypassed queue. The repair is the four-way split and a page list that is shorter than the company's anxieties.

Two accountable owners, so that both feel included. The definition owner and the pipeline owner are both marked A "because it is a partnership." When the number moves, each partnership member waits. The repair is one A and a mandatory consult. Partnership is the consulted column plus a shared incident note. It is not a shared A.

On-call as heroics. The rotation is expected to rewrite SQL, patch the player, and answer the executive channel. They will do it once. The repair is a severity table whose overnight actions are revert and replay, and a consumer experience that tolerates a labeled stale snapshot.

Embedded engineers, private platforms. Every squad picks an orchestrator and a bucket policy. Prohibited fields are enforced in the squad that remembered. The repair is a hub that owns the paved path, and a rule that spokes do not get cloud-admin credentials. Autonomy is the metric spec, not the account boundary.

Central team as a service desk. All domain knowledge arrives as tickets titled "add this column." The column is added. The grain breaks a month later. The repair is to move definition accountability to people who sit with the decision, and to let the central team refuse tickets that have no spec version.

Grants by screenshot. A credential is copied into a notebook during an incident and never revoked. The second dashboard is now a production system with no owner. The repair is time-limited access, a request artifact, and a periodic query for roles that can write or that can read raw data without a matrix row.

The architect who does not attend reopenings. A decision record says the stream will be adopted when a second consumer appears. The second consumer appears. The architect is in a naming workshop. Spokes build the stream privately. The repair is to put the architect, or the acting owner of that decision, on the review that the reopening condition triggers. A role that never sees the consequence is a commentator.

Troubleshooting and Design Review

Symptom: An incident channel spends the first hour deciding who owns the defect.

Likely cause: the four ownership questions were never written, or they live in a slide from last year.

Investigation: identify whether the wire, the spec, the job, or the grant changed. Look for a second writer: a personal schema, an extra credential, a BI model with its own SQL.

Corrective action: restore the last good publication if the official table moved incorrectly. Assign the defect to one class. Update the matrix this week, not after the postmortem cycle rewrites it into folklore.

Symptom: The publication is on time and executives still quote a different number.

Likely cause: a second product with no owner, often a personal dashboard or a spreadsheet export.

Investigation: ask for the query. Find the credential it uses. Compare definition versions.

Corrective action: revoke or redirect. Inform the Monday audience which dataset is official. Do not change v3 overnight to match the unofficial number. If v3 is wrong, the definition owner schedules a version bump and a backfill plan.

Symptom: Product engineering ships a payload change and the raw zone rejects a full day.

Likely cause: the event contract was not a joint artifact. Data engineering was informed by the release notes, which is the wrong column.

Investigation: diff the payload against the checker. See whether a consult was required on the pull request.

Corrective action: quarantine stays failed. There is no "accept anything" override without the governance and data engineering roles. Add the contract test to the player pipeline. Replay the day after the payloads match. The late publication is the consequence of a skipped consult, and it should be visible.

Symptom: The platform queue is three weeks long, and a squad stands up a second warehouse.

Likely cause: hub responsibilities include domain metric work, or the spoke has no path to ship a modeled table on the shared engine.

Investigation: classify the queued tickets into engine work and definition work.

Corrective action: move definition work to the spoke, give the spoke a paved project with no admin rights, and keep the hub on grants and upgrades. A second warehouse is an ownership escape. Close it before it has a consumer you cannot move.

Symptom: The same person is paged for player outages and for mart delays, and both responses are shallow.

Likely cause: one human, two hats, one undifferentiated rotation.

Investigation: count pages by class for a month.

Corrective action: split the rotations even if the backup list overlaps. The data page's runbook starts at publication and watermark, not at the player deploy. Hire or narrow the promise when Sev-2 pages exceed what the backup can restore. Do not respond by declaring a follow-the-sun rotation you do not staff.

A design review of a responsibility map asks: Where is the single A for this decision? What is the artifact? What is the page condition, and what is explicitly not a page? Which grants would let a non-owner write? What team shape are we in, and what measurement changes it? If the answer to the first question is a team name, the review stops.

Interview Questions and Answers

Q1. What is the difference between responsible and accountable for a dataset?

Responsible people do the work. Several can share it. Accountable is the single role that answers for the outcome and can stop a change. For the completion publication, the data engineer is both, in our small-team assumption: they run the job and they answer for it landing. For metric wording, analytics is accountable and may still be responsible for the draft, while the data engineer is responsible for the replay and is consulted on feasibility. If both are marked accountable, an incident spends its first hour on the boundary. I will not ship a matrix with two A's on one decision.

Q2. Who owns data quality?

Nobody, as a single blob, and several people in specific classes. Product engineering owns whether the event that was emitted matches the contract. The data engineer owns whether accepted events landed once and whether the job published. The definition owner owns whether the SQL matches the spec. Governance owns the prohibited-field list and retention. An analyst often discovers a semantic break and does not thereby become the owner of the pipeline. I route a defect to one of those classes before I open a tool.

Q3. How do you staff on-call for a two-person data function?

With a short page list and a restore-only backup. Pages are missed publications, contract rejections above the threshold, and prohibited-field detections. They are not metric debates or player outages. Overnight action is replay of an accepted spec or revert to the last publication. The serving layer must show the last good timestamp so a miss is not displayed as zero. I do not draw a follow-the-sun rota for two people in one zone. If the promise cannot be met by that rotation, I narrow the promise or I staff it. I do not pretend the handbook's rotation exists.

Q4. When would you move from a centralized data team to hub and spoke?

When a second domain product needs a different pager and a different definition owner, and a single queue is already causing squads to build side pipelines. The hub keeps engines, identity, CI, and the paved path. Spokes own domain specs and jobs. I would not embed people into squads with no hub while prohibited-field rules are still being reinvented. The trigger I write down is a second production publication with a different consumer, or a ticket age that is pushing domains off the platform. A reorganization without that trigger copies an org chart from a larger company.

Q5. How should an event-contract change be approved?

Product engineering is accountable for the wire, because they generate it. Data engineering is consulted and holds the checker: a change that the contract module rejects does not ship. Governance is consulted when fields appear or when identifiers change meaning. Analytics is informed unless the change alters a metric's grain, in which case the definition owner is consulted and the spec version bumps in the same release. The artifact is a versioned contract and a green test, not a meeting note. Shipping the player first and "fixing the pipeline next week" is a skipped consult. The raw zone should reject the day rather than accept a null attempt number.

Q6. A stakeholder says the data team is blocking a dashboard. What do you look at?

Whether the request is a new cut of a published mart or a new definition. A new cut is an analyst's job on the existing grain, and a long queue means the central team has been inserting itself into serving. A new definition needs a spec, a version, a backfill plan, and one accountable owner. I also look for a side pipeline the stakeholder has already started, because "blocking" sometimes means "refused to bless an unowned extract." The repair is to make the paved path faster for legitimate cuts and to keep the refusal for unowned products.

Q7. Why is a shared warehouse password an ownership defect?

Because it creates writers and readers the RACI does not name. Someone will repair a day by hand, or copy a prohibited column because the join is easy, and the publication job will no longer be the only path. I want the job role to be the only writer of the mart, analysts to select published snapshots, and raw storage to be inaccessible except through the extract and the job. Access requests have a purpose and an expiry. A password in a notebook is an unowned production system.

Q8. How do data engineers and analytics engineers divide a small team?

I divide hats, not headcount, until the company hires the second hat. The analytics hat is accountable for metric wording, grain in the spec, and accepting a version. The data-engineering hat is accountable for ingestion, replay, failure, and publication. The same person may wear both on different days and must not approve their own spec change without the other consult: in a one-person case, a second reviewer is the analyst or the product manager on the wording, and the platform engineer on the grants. Pretending the company has a full analytics-engineering organization when it has one generalist produces a RACI nobody can execute.

Q9. What should the instructor see if the mart job fails?

The last published snapshot, the timestamp, and the definition version. Not a zero, not a spinner that retries against the operational database, and not a number an on-call engineer typed into a channel. The studio engineer is responsible for that display. The data engineer is accountable for the publication promise and for specifying this failure behavior. A failure that looks like "completions disappeared" is a serving bug on top of a pipeline bug. The RACI has a row for it because the pipeline row alone does not protect the instructor.

Q10. How does Conway's law show up in a data platform?

The platform will grow a writer for every group that can change it. Four squads with production credentials produce four definitions. A hub that is the only group allowed to create projects, plus spokes that can only ship through a paved path, produces one publication per product. I treat org charts as architecture constraints. If leadership wants embedded autonomy and also wants one completion rate, the autonomy has to be the spec, not the storage account. Otherwise the org chart and the metric will disagree, and the metric will lose.

Q11. Who is allowed to change a published historical day, and who is only informed?

The data engineer accountable for the pipeline may replace a day by replaying an already accepted definition version, and records the publication. Changing the definition that applies to that day is a new version, accountable to the analytics hat, with a backfill plan, and the Monday audience is informed before they use the number. A scientist who needs a different label for a model does not get to overwrite the official day. They get a feature contract and, if necessary, a separate table. Finance, once they depend on the mart, are consulted before a restatement outside the agreed window, and they hold a veto if the day is closed. Until they are consumers, they are not in the row.

Q12. What evidence would show that a responsibility map is working six months later?

Incidents state an ownership class in the first note. Publication misses page the data rotation and do not page the analyst. There is still exactly one writer role on the mart. Personal extracts created during old incidents are gone. Contract changes are blocked in CI when the checker fails. A second domain, if it appeared, uses the hub's job runner rather than a new VM. The matrix in git matches the grants you can list in the database. If the only evidence is that people are busy and helpful, the map is a social habit and it will fail on the next vacation.

Hands-On Exercises

Exercise: Write a responsibility map for a data platform team

Problem statement. You are staffing the Northline Marketplace analytics estate from lesson 2: one completion product, one payout product, a platform hub, and a single governance owner. The company has four engineers who can take data work, not fourteen. Produce a RACI they can run.

Tasks.

1. List the decisions: event contracts, each publication, each metric wording, grants, prohibited fields, studio or dashboard staleness, and the payout restatement.
2. Assign exactly one accountable role per decision. Mark any hat that is worn by the same human as another hat, and name the second reviewer that prevents self-approval.
3. Write page conditions for pipeline rows and a "does not page" list of at least four items.
4. Choose centralized, embedded, or hub-and-spoke. Justify it with the number of pagers. State the measurement that would change the shape.
5. Specify writer and reader grants for the completion mart, the payout fact, and the raw zone.
6. Draw the incident flow for a missed completion publication while payouts are healthy.
7. Add one Conway risk you are explicitly refusing, such as a private orchestrator or a shared warehouse password.
8. Implement the matrix check from this lesson, or an equivalent test, against your rows. Include one row that must fail.

Expected deliverables. The matrix, the page list, the grant notes, the diagram, and the test output.

Evaluation rubric.

You pass when a reviewer can pick any incident in the list and name one accountable hat without using the phrase "the data team."

Lesson Summary

Roles are hats with artifacts, not titles on a hiring plan. A data product needs a single accountable owner for its definition, a single accountable owner for its publication, a product owner for the generating contract, a platform owner for grants and engines, and a governance owner for fields that must not land. Those hats may sit on a few people. They may not sit on "everyone."

The TechLearningPro completion incident that survives a green job is almost always a second writer or a second definition with no row in the matrix. The repair is a RACI stored beside the pipeline, grants that match it, a page list a two-person rotation can carry, and a serving path that shows the last good publication while the humans sort the class of defect.

Team shape follows the number of domains you actually operate. Centralized is honest for one mart. Hub and spoke becomes honest when a payout pager appears. Embedded without a hub copies the org chart into a swamp. Lesson 4 starts the language those owners will review: Python that makes the contract and the matrix executable, with the typing and the failure behavior a pipeline has to show before it is allowed near the raw zone.

Further Reading

• Melvin Conway, "How Do Committees Invent?" — the original observation that systems copy communication structure. Read it before you draw an autonomous squad per dataset.
• Project Management Institute, practice guides on a responsibility assignment matrix — the RACI letters. Use the letters as an incident tool, not as a template to fill every cell.
• Joe Reis and Matt Housley, Fundamentals of Data Engineering — stakeholder and role chapters, useful for the split between generation and the analytical estate.
• Google's Site Reliability Engineering — on-call, severity, and the idea that an overnight response has a bounded action. Lesson 62 turns this into data SLOs. The bounded action belongs in the role design now.
• PostgreSQL 16 roles and privileges — the grant model the matrix has to match. Primer: /learn/postgresql.
• Continue this course: /learn/data-engineering/python-fundamentals-for-data-engineering. The language primer, if the dataclasses in this lesson were unfamiliar, is /learn/python.