Staff Engineer Architecture Decision-Making

Advanced System Design & Software Architecture · Lesson 61 · Architecture Practice, Interviews and Capstone

Lesson Overview

TechLearningPro's enrollment path called two downstream systems inside the same database transaction that inserted the enrollment row. One call asked catalog whether the lesson was published. The other asked the email vendor to send a receipt. On a Tuesday in enrollment week the vendor's p99 climbed past eight seconds. HikariCP filled, enroll requests piled up behind it, and the classroom home page — which shares the pool — started returning 503. A senior engineer arrived at the incident review with a diagram: a broker, five consumers, and the sentence "we should stop doing synchronous calls."

The sentence named a mechanism. It did not name a decision. Catalog's answer changes what the learner is allowed to see in the HTTP response. The email vendor's answer does not. Replacing both with events would have made enroll faster and would also have made "you are enrolled" a guess. The staff job in that review was to write the question the room was actually allowed to answer, put the status quo on the table as a real option, and record who could veto the result.

This lesson is the practice of that job. Architecture Documentation and ADRs already specifies the record: status, context, drivers, options, decision, consequences. Here the work is everything that has to happen before a record deserves to be accepted. You frame the decision so it has one owner and a time box. You write options that include doing nothing and doing less. You attach each option to a quality scenario from Architectural Thinking and Quality Attributes, including the tax. You separate an internal proposal — an engineering RFC — from the decision record that survives the meeting. You name stakeholders, including the people who were not in the room and still operate the pager. You classify the choice by how expensive it is to undo, because a reversible experiment and a one-way data-shape change do not deserve the same ceremony.

The sequence in High-Level System Design Methodology tells you what a design packet contains. This lesson tells you how a staff engineer gets a contested packet through a room of people who each want a different box. The worked decision is the enrollment integration: which calls may leave the request path, and which consistency the learner is still owed.

Learning Objectives

By the end of this lesson you will be able to:

1. Frame an architecture decision as a single question with an owner, a time box, a success test, and an explicit list of what the decision will not settle.
2. Write at least three options, including the status quo and a smaller change, each tied to a quality scenario and a cost the sponsor can see.
3. Classify a choice as reversible or one-way, and match the amount of review to that classification.
4. Use an internal RFC as a proposal for comment and an ADR as the decision, and refuse to treat a merged RFC as if the decision had already been made.
5. Map stakeholders by the kind of input they own — product behavior, on-call load, security boundary, money — and record dissent without requiring unanimity.
6. State a technical strategy as a standing decision that makes the next ten local choices cheaper, with a condition that reopens it.
7. Facilitate the review so the output is a decision and a consequence, including the option refused, rather than a longer diagram.
8. Review someone else's proposal by asking which call still has to answer inside the user's request, and which failure domain the new mechanism adds.

Prerequisites

• Architectural Thinking and Quality Attributes: trade-offs are scenarios with response measures, not adjectives. A decision that "improves scalability" with no measure is not ready.
• Requirements and Architecture Drivers: drivers are the inputs. A preference dressed up as a driver wastes the review.
• Layered and Modular Monolith Architecture and Event-Driven Architecture Fundamentals: the enrollment decision sits on top of the modular monolith and the outbox. This lesson does not re-teach either.
• Architecture Documentation and ADRs: you will point at that record shape. You will not invent a second template.
• High-Level System Design Methodology: behavior, assumptions, and failure notes are how an option becomes specific.
• Message Queues and Event Streaming, Enterprise Integration Patterns, and Architecture Migration and Modernization: the mechanisms an RFC may propose, and the migration tax of adopting one.
• Comfort reading a one-page proposal. The Java in this lesson is a port and an outbox insert using Spring's @Transactional boundary and JdbcClient. It introduces no messaging client.

Detailed Core Concepts

What makes a decision a staff decision

Most pull requests are local. They change a module the author operates, they can be reverted by reverting the commit, and they do not teach the next team a new default. A staff decision has at least one of these properties:

• It crosses a team boundary, so the people who feel the consequence do not all report to the person who wants the change.
• Undoing it costs more than doing it, because data, consumers, or a public contract will exist afterward.
• It sets a precedent. The next squad will copy it when they are tired, so the written reason matters more than the mechanism.
• It spends a scarce budget: on-call attention, a security review slot, a migration window, or an error budget you have not met yet. Error budgets return in Non-Functional Requirements and SLO Design. Here you only need to notice that "ship the broker this week" is a budget claim.

Senior engineers often make these choices anyway, inside a design document that starts with a solution. The staff contribution is to slow the start by one page. Will Larson describes staff work as the work that remains when no single team owns the outcome. The practical test in a review is smaller: if two teams can each implement their preferred option without talking, and the company still functions, you do not have a staff decision. If one team's "small improvement" changes the other team's pager, you do.

Framing is the decision before the decision

A frame is the question, plus the constraints on how it may be answered. A bad frame is "should we use Kafka?" That question can be answered by taste. A frame this course will accept has five parts.

The question is one sentence a stranger can disagree with. "Which downstream calls inside POST /enrollments may complete after the HTTP response, and what state is the learner allowed to observe in that gap?" The question names an operation, a boundary, and the user-visible fact at stake.

The owner is the person who will accept the record. Consultation is not ownership. If catalog, enrollment, and notifications all "own" the decision, nobody schedules the follow-up when the consumer lags. One owner can still be required to obtain a veto from security or from the on-call lead. Those are constraints on the owner, not additional owners.

The time box is when the decision will be made with the information then available. Architecture reviews expand to fill the anxiety in the room. A two-week RFC with a decision meeting on a named day beats an open-ended "architecture thread." Information that cannot arrive inside the box is written down as an assumption, in the same sense lesson 23 uses assumptions. It is not a reason to skip the decision.

The success test is how you will know the decision worked, in production, by a date. "Enroll p99 at the API stays under 400 ms while the email vendor injects five seconds of delay, and a receipt still arrives within fifteen minutes for 99 percent of commits." If you cannot write the test, the options are still slogans.

The non-goals keep the meeting from annexing the platform. "This decision does not choose a company-wide broker, does not change how payments capture, and does not let catalog become eventually consistent for the published-or-not fact." Non-goals are how a staff engineer stays inside the problem that hurt the user.

Richard Rumelt's useful cut for strategy — diagnosis, guiding policy, coherent action — scales down to a single frame. The diagnosis is the pool exhaustion caused by a call the user does not need synchronously. The guiding policy is "a call stays on the request path only when its result is part of the response or the transaction's truth." The coherent action is the option you pick and the work you sequence. A frame that skips the diagnosis and starts at the broker has no policy. It has a purchase.

Options are incomplete until the status quo is one of them

A proposal with one option is a request for permission. A review needs a real set:

1. Status quo. Leave the calls where they are. Write the pain in numbers: vendor delay, pool wait, user-visible failure rate. The status quo is allowed to win when the alternatives cost more than the pain, or when the pain is an incident you can fix with a timeout.
2. The smallest change that meets the success test. Often a timeout, a bulkhead, or moving one call. Circuit Breaker, Retry and Bulkhead Patterns is the vocabulary for isolating a dependency without a new integration style.
3. The structural change people actually came to argue for. Event-carried integration, a new service, a new store.
4. The option you include so the structural change has to beat something serious. For enrollment, that option is "outbox for mail, synchronous read for catalog," which uses a mechanism the codebase already operates.

Each option states, in the same order: what the user observes, what is durable at the moment of the HTTP response, what new failure exists, what operational work appears, and what you would need to undo it. An option that describes only the happy path is not an option yet. Lesson 23's failure note — how we notice, what the caller sees, what is persisted, what we refuse to do automatically — is the right attachment for every option, not only for the winner.

Fairness to the losing option is part of the writing. If you describe events as "modern" and the synchronous call as "legacy," you have written a campaign. Describe the synchronous catalog read as "the response includes a fact catalog still owns, fetched under a timeout of 80 ms, failing closed if catalog does not answer." That description can lose on latency and still be the right description. Future readers need to see why it lost, or they will revive it without the timeout.

Trade-offs are paired movements, not a score

A trade-off is a single choice that improves one response measure and worsens another. Lesson 2 calls the place this happens a trade-off point. In a staff review you put the pair in a row the sponsor can read:

The table does not contain a column called "score." Weighted scores feel decisive and hide the stakeholder who cares about the one row that scored badly. The owner looks at the success test and at the vetoes. Security may veto an option that puts a learner email address on a bus several teams can read. Finance may veto a second region that this decision does not need. The on-call lead may veto a new deployable with no owner. Those vetoes are inputs. They are written next to the option, not discovered after the ADR is "accepted."

Reversibility decides the ceremony

Jeff Bezos's 2015 shareholder letter distinguishes type 1 decisions, which are hard to reverse, from type 2 decisions, which are reversible experiments. The letter is a management essay, not an architecture standard, and the distinction still earns its place because teams apply one ceremony to both kinds and get both wrong. They hold a six-week review to change a timeout, and they pick an event schema in a standup because "we can always add fields."

Architecture has its own reasons a door becomes one-way:

• Data shape. A primary key, a tenant identifier embedded in every event, or a column that downstream reports group by. Design a Multi-Tenant SaaS Platform shows how early tenant placement becomes permanent. Adding a field is easy. Changing the meaning of an existing field after consumers persist it is a migration program, which lesson 48 treats as its own project.
• External promise. A public API, an SLA credit clause, a partner who built on your webhook order. API Design, Versioning and Contracts is the contract half of this door.
• Operational dependency. Once another team pages on your topic, you cannot delete the topic because you changed your mind. The consumers are the irreversibility.
• People and process. A new on-call rotation, a compliance scope, a second language in production. These reverse slowly even when the code reverts.

A reversible choice can ship behind a flag, with a date to delete the flag and a metric that decides the deletion. A one-way choice gets the full frame: options, dissent, a rehearsal of the undo path written while the undo path still exists. "We can rewrite it later" is not an undo path. An undo path names the data movement, the consumer cutoff, and the week of engineering.

The middle is the two-way door with a timer. The outbox for mail is reversible until a second product team starts reading the same table to build a CRM feed. The RFC should say that. The moment a second consumer appears, the decision class changes, and the owner reopens the record before agreeing to the feed. That sentence is cheaper than a platform migration two years later.

An RFC proposes. An ADR decides.

Lesson 10 owns the ADR. This lesson owns the proposal that comes first, because many organizations let a document titled RFC become the decision by accident.

An internal engineering RFC is a request for comments. The name is borrowed from the IETF's habit of writing a proposal down so strangers can object. It is not an IETF RFC, it has no standards-track status, and it should not wear RFC numbers that collide with the real series. Call it RFC-enroll-mail-outbox or a pull-request title. The document's job is to expose the frame, the options, and the author's preferred option to people who were not in the author's head. Commenters are supposed to change it. A proposal that cannot survive comments was not ready to be a decision.

An ADR is written when the owner decides. It may cite the RFC. It does not require the RFC to have reached unanimity. IETF RFC 7282 describes "rough consensus and running code": consensus is not a vote, and it is not the absence of dissent. The chair looks for unresolved technical objections, not for a pile of agreements. An architecture review can steal that test. Unresolved means "this objection names a broken success test or a veto the owner does not have authority to ignore." "I dislike brokers" is dissent to record, and it is not unresolved if the success test is met another way.

Practical sequence:

1. The author opens the RFC as Proposed, with the frame and options, and with their preference marked as a preference.
2. Commenters have a window. The author revises in the open, or answers why an objection did not change the text.
3. The decision meeting lasts as long as the frame. It ends with an owner statement: accept option C, reject B for this reason, reopen if that measurement appears.
4. The ADR is merged as Accepted, linking the RFC. The RFC's status becomes Decided — see ADR-214. The RFC is not deleted. Future readers need the objections.
5. Implementation links the ADR. A pull request that builds the rejected option fails review because the link is wrong, not because a staff engineer remembers the meeting.

Superseding works the way lesson 10 already said. You do not edit the old decision into its opposite. You write a new record that points at the old one and says which assumption died.

Stakeholders are defined by the fact they can break

A RACI chart with fifteen names is a distribution list. A staff map is shorter. For each stakeholder, write the fact they are accountable for, the input you need from them, and whether they hold a veto.

People who are "informed" still get the ADR. They do not get a blocking seat. The failure mode of consensus culture is that the most available calendar wins and the on-call lead reads the document during the incident. Send the RFC to the pager owner with a named question: "Will you accept a fifteen-minute mail lag as healthy?" Their silence inside the window is not consent. Their absence after a direct question and a reasonable wait is recorded, and the owner either delays or explicitly accepts the risk of deciding without them.

Disagree and commit is a commitment by the dissenters to implement the decision as written, and a commitment by the owner to write the dissent into the consequences. It is not a request that people pretend they agreed. The value of the written dissent is that the reopen condition often comes from it. The catalog owner who says "I will not support a ten-minute stale published flag" has just written the trigger for a future record.

Technical strategy is a standing decision

A single ADR is a tactic. A technical strategy is the small set of standing decisions that tell a team which tactics are in bounds for the next year. Rumelt's test applies: a strategy has a diagnosis of the company's actual constraint, a guiding policy, and actions that match the policy. "Be a platform" is not a policy. "Commerce remains a modular monolith until a compliance boundary or a separate on-call team forces a deployable split" is a policy. Lesson 10 already recorded that one. Strategy work at staff level is keeping the next RFC consistent with it, or reopening it on purpose.

Good strategy also says what you will not do. TechLearningPro can refuse company-wide choreography for request/response reads. It can refuse a second language in the enroll path this year. It can refuse to let new product features add synchronous vendor calls inside a transaction. Those refusals are the strategy. They make local design cheaper because the author of the next RFC starts from "mail leaves the transaction; facts required for the response do not," and only spends the room's time if they need an exception.

Strategy dies when it is a slide disconnected from ADRs. The way you keep it alive is boring: every accepted ADR cites the strategy sentence it follows, or cites the sentence it is amending. Once a quarter, the staff engineer reads the exceptions. A pile of exceptions means the strategy is wrong or the review is weak. Either finding is useful. A strategy with no exceptions and no recent ADRs is usually a poster.

Facilitation is part of the design

The best frame still fails in a meeting that re-draws the diagram for an hour. A staff engineer runs the room with a few rules.

The RFC is the pre-read. The meeting does not start by the author narrating it. It starts with objections that survived the comment window. If there are none, the owner decides in ten minutes and the meeting is a success.

One decision is in scope. A new concern is written on a list with an owner and a date. It does not amend today's question unless it is a veto.

The author speaks for the options, including the ones they dislike, before arguing for a preference. This is the only reliable way to discover that the author does not actually understand the status quo.

The last five minutes produce sentences, not nods. The owner reads the decision, the rejected option, and the reopen condition. Someone else types them into the ADR before the meeting ends. Memory is not a record.

You will still leave some reviews without a decision. That is allowed when a veto holder found a hole in the success test. It is not allowed because the room enjoyed the discussion. Schedule the continuation before you adjourn, with the missing fact named.

Architecture and Design Diagrams

The decision flow is the artifact. Components show up only after an option survives it.

mermaidCopy
flowchart TD
    inc[Incident or proposal] --> frame[Write the frame: question, owner, time box, success test, non-goals]
    frame --> opts[Options: status quo, smallest change, structural change]
    opts --> rev{Reversible?}
    rev -->|Yes, inside one team| exp[Time-boxed experiment and a flag]
    rev -->|One-way or cross-team| rfc[RFC for comment]
    exp --> adr[ADR with reopen condition]
    rfc --> meet[Decision meeting on objections]
    meet --> veto{Unresolved veto?}
    veto -->|Yes| revise[Revise the frame or reject]
    veto -->|No| adr
    adr --> impl[Implementation links the ADR]
    impl --> check[Success test on production measures]
    check -->|Met| keep[Leave the record accepted]
    check -->|Missed| reopen[Supersede; do not edit history]

Stakeholders enter at the RFC, not at implementation. The map for the enrollment decision:

mermaidCopy
flowchart LR
    subgraph decision [Decision]
      owner[Enrollment owner]
    end
    subgraph veto [Veto on a named fact]
      cat[Catalog freshness]
      page[Notifications on-call]
      sec[Security data class]
      prod[Product promise]
    end
    subgraph consult [Consulted, not blocking]
      sup[Support wording]
      fin[Cost of a new broker]
    end
    cat --> owner
    page --> owner
    sec --> owner
    prod --> owner
    owner --> sup
    owner --> fin

The preferred option's runtime, once the decision exists, is deliberately small. Catalog remains a synchronous read. Mail becomes a row in an outbox committed with the enrollment. A worker the notifications team already runs delivers it.

mermaidCopy
sequenceDiagram
    actor Learner
    participant API as Enrollment API
    participant DB as Enrollment DB
    participant Cat as Catalog
    participant Worker as Mail worker
    participant Vendor as Email vendor
    Learner->>API: POST /enrollments
    API->>Cat: isPublished(lessonId) under 80 ms
    alt catalog says no or times out
        API-->>Learner: 409 or 503, no row
    else catalog says yes
        API->>DB: insert enrollment and outbox row, one transaction
        API-->>Learner: 201 Enrolled, receipt pending
        Worker->>DB: claim outbox row
        Worker->>Vendor: send receipt
        Vendor-->>Worker: accepted
        Worker->>DB: mark sent
    end

The sequence is the consequence of the decision. It is not the decision. A review that starts with this picture will spend its time naming the broker the picture deliberately does not have.

Technical Explanation

How to write an option so a dissenter can use it

An option has a name, a mechanism, a user-visible behavior, a consistency claim, a failure mode, a cost, and an undo note. Six sentences is enough. The mechanism sentence uses words the organization already defined. "Publish EnrollmentCommitted to the existing outbox table" is a mechanism. "Go event-driven" is a mood.

Consistency has to be specific because CAP Theorem and Consistency Models taught you that "strong" and "eventual" are not designs. For mail, the claim worth writing is: the enrollment row and the outbox row commit together; the vendor may be minutes behind; a crash after commit and before send retries from the row; a duplicate send is possible and the template must tolerate it. That is an at-least-once handoff with a single-transaction local claim. Idempotency and Exactly-Once Processing is where you go if someone in the review says "exactly once." You do not promise it in the RFC.

For the catalog read, the claim is different: the response's allow-or-deny uses catalog's current answer, with a timeout, and a timeout fails closed. There is no local cache in this option. If product later accepts sixty seconds of staleness, that is a new decision, because it changes a veto holder's fact. Write the absence of a cache as part of the option so a helpful follow-up pull request does not add one quietly.

Why the synchronous call was in the transaction in the first place

The original author put the vendor call inside the transaction for a reason you should respect. They wanted "no enrollment without a receipt" as a single atomic outcome. The desire is a product sentence. The mechanism — holding a database transaction open across a network call — buys that sentence by risking every other user of the pool. Database Design and Indexing for Scale assumes transactions are short. A transaction that waits on SMTP is a connection-pool incident wearing a correctness costume.

The staff move is to split the product sentence into two promises that can have different mechanisms. "No enrollment row exists unless we have taken responsibility for a receipt" can be the outbox. "The vendor has accepted the message before the user sees success" is a stronger promise, and it is the one that caused the outage. Ask product which sentence they bought. Often they bought the first one and the code implemented the second because it was easier to explain in one method.

The catalog call is the opposite shape. The user is waiting for a yes or no that another context owns. Domain-Driven Design Fundamentals gives you the language: enrollment does not own "published." Copying it into an event projection creates a second writer in practice, even if catalog remains the source, because the projection will be treated as authoritative the first time catalog is down. That is how ownership erodes. Keep the synchronous read until a scenario forces the copy, and write the scenario's staleness into the record.

Coupling you trade, not coupling you delete

Moving mail to an outbox removes runtime coupling: enroll no longer fails when the vendor is slow. It adds contract coupling: the outbox payload becomes a schema someone else codes against. It adds operational coupling: lag on that table is now a user-visible debt, because the receipt was promised within fifteen minutes. Hohpe and Woolf's enterprise integration patterns, which lesson 47 works through, are a catalog of these trades. A staff engineer names the trade in the RFC so the room does not hear "decoupled" and stop listening.

There is also team coupling. If enrollment writes the outbox and notifications owns the worker, you need an agreement on the payload, the retry policy, and who pages when the oldest unsent row exceeds fifteen minutes. That agreement is a small contract. It is healthier than a shared database transaction, and it is still a dependency. Put the owner of the lag alarm in the RFC. A design whose alarm has no owner has not left the whiteboard.

Reversibility in the schema, worked as arithmetic of pain

Suppose the outbox payload includes learnerEmail because the worker should not call back into enrollment. Security's data-class note says email is personal data and the table is readable by anyone with production database access, which today includes analysts. Two responses exist.

One: the payload carries enrollmentId only, and the worker reads email through an API that authorizes the worker's identity and writes an audit entry. The undo cost stays low. The worker gains a runtime dependency, which you already know how to time-bound.

Two: the payload carries email, the table is restricted, backups are encrypted, and a retention job deletes sent payloads after seven days. This can be legitimate. It is a heavier decision because every backup already taken contains the column, and a bad retention job is a privacy incident. Architecture Security and Threat Modeling is the method for that review. In this lesson the point is narrower: the presence of the column changes the decision from type 2 to type 1. Call that out in the RFC before someone adds the column "just for debugging."

What the owner writes in the consequences

Lesson 10 asks for consequences that include the cost of the option you accepted. For this decision the consequences read as obligations:

• Enroll no longer rolls back when mail fails. Support's macro changes on the same day the code ships.
• A receipt can be late or duplicated. The template says "you are enrolled" in a way that a second copy does not alarm the learner.
• Catalog timeouts become visible as 503 on enroll, which is the remaining synchronous dependency. That is accepted.
• No new broker is introduced. A later RFC that wants a broker for this flow has to explain what the outbox cannot do.
• Reopen the record if mail lag exceeds fifteen minutes for more than one percent of rows in a week, or if a second consumer wants the outbox.

Obligations are consequences. "Improved resilience" is not a consequence. It cannot be checked, and it cannot be violated, so it cannot govern a pull request.

Rough consensus without a vote

Counting hands feels fair and selects for the people who could attend. The owner instead lists the objections that remain and classifies each one:

• Incorporated. The payload dropped the email address. The RFC changed.
• Overruled, with a reason. "Use a broker because it is the standard." Overruled because the success test is met by the outbox, and the strategy says we do not add infrastructure for one flow. The dissent is in the ADR.
• Unresolved veto. Catalog has not agreed that "published" can be stale, and option D required that agreement. Option D is not selectable. This is not a failure of courage. It is the frame working.

If every objection is a restatement of taste, the owner decides. If a veto holder cannot be reached about a fact only they know, the owner waits or narrows the decision so it does not depend on that fact. Narrowing is a skill. "We will not build the projection this quarter" lets the mail outbox proceed while catalog's question stays open.

Practical Implementation or Design Walkthrough

Problem statement

Enrollment week showed that POST /enrollments shares its database pool with the rest of classroom traffic and holds a connection across an email-vendor call. Catalog's published check is also on that path. Product still wants the learner to leave the request knowing whether they are enrolled. Support wants a true sentence when the receipt is late. Notifications will operate delivery if the contract is small. Nobody is asking to redesign commerce.

Requirements, assumptions, constraints

Behavior in scope: a learner enrolls in a published lesson and eventually receives one receipt. Behavior out of scope: waitlists, payments, catalog authoring, a company-wide event bus.

Assumptions, labeled so they can be replaced: peak enroll is 20 requests per second; the vendor's normal latency is 200 ms and its bad-day latency is multiple seconds; catalog's normal latency is 15 ms; a fifteen-minute receipt delay is acceptable to product if the HTTP response is honest; duplicate mail is acceptable if the copy is idempotent in wording.

Constraints: the identity provider and the catalog ownership rules stay as they are; learner email is personal data; the team will not add a new deployable this quarter; the standing strategy is the modular monolith plus the outbox pattern from lesson 9.

Success test: with the vendor delayed by five seconds in a staging fault injection, enroll API p99 stays under 400 ms, no enrollment transaction waits on the vendor, and at least 99 percent of committed rows have a sent receipt within fifteen minutes.

The RFC, condensed

Question. Which calls inside POST /enrollments may finish after the response?

Preference. Option C below. The preference is not the decision.

Option A — status quo. Vendor call inside the transaction. Fails the success test. Matches the strongest mail promise. Undo cost is zero because it is already built.

Option B — bulkhead and short timeout, call stays synchronous. Enroll fails when the vendor is down, so the learner is not enrolled and will retry, possibly double-submitting unless the idempotency key from lesson 37 is already in place. Meets a weaker test ("we fail in under 400 ms") and misses "receipt still happens."

Option C — outbox for mail, synchronous catalog read, fail closed. Meets the success test. Changes the promise from "vendor accepted" to "we committed to send." No new broker.

Option D — catalog projection and asynchronous everything. Meets latency even when catalog is down. Requires catalog to accept staleness and requires a rebuild path. Rejected inside this time box because the veto is not available and the incident did not involve catalog correctness.

Interfaces

The application port is local. It is not a vendor SDK.

javaCopy
public interface ReceiptScheduler {
    void schedule(ReceiptRequest request);
}

public record ReceiptRequest(String enrollmentId, String lessonId, Instant enrolledAt) {}

The HTTP contract gains an honest body. 201 means the enrollment row committed. The receipt status is PENDING, not SENT. Clients that treated 201 as "the inbox already contains the mail" were coupled to the accident. Release notes say so.

Catalog stays a blocking client with a deadline:

javaCopy
public interface CatalogPublished {
    /** Completes within 80 ms or throws CatalogUnavailable. */
    boolean isPublished(String lessonId);
}

The transactional write

The enrollment insert and the outbox insert share one Spring transaction. JdbcClient is the JDBC facade in current Spring Framework. The SQL is ordinary PostgreSQL.

javaCopy
@Transactional
public Enrollment enroll(String learnerId, String lessonId, String idempotencyKey) {
    if (!catalog.isPublished(lessonId)) {
        throw new LessonNotPublished(lessonId);
    }
    String enrollmentId = UUID.randomUUID().toString();
    jdbc.sql("insert into enrollment(id, learner_id, lesson_id, idempotency_key) values (?, ?, ?, ?)")
        .params(enrollmentId, learnerId, lessonId, idempotencyKey)
        .update();
    jdbc.sql("insert into outbox(id, type, payload, created_at) values (?, 'receipt', ?::jsonb, now())")
        .params(UUID.randomUUID().toString(), json(new ReceiptRequest(enrollmentId, lessonId, Instant.now())))
        .update();
    return new Enrollment(enrollmentId, ReceiptStatus.PENDING);
}

A unique constraint on idempotency_key keeps retries from creating a second seat. The worker claims rows with FOR UPDATE SKIP LOCKED, which PostgreSQL supports, so two workers do not send the same row. The claim, the vendor call, and the sent_at update are three steps, not one transaction held across the vendor. A crash after the vendor accepts and before sent_at is written causes one duplicate mail. That duplicate is in the failure note, and the template does not say "action required."

Failure handling

Validation

• A test stubs the vendor with a five-second sleep and asserts the request thread returns in well under a second and that the outbox row exists.
• A test rolls the transaction back when the enrollment insert fails and asserts no outbox row.
• A test runs two workers against one row and asserts one send.
• A staging fault injection is the success test, not the unit test. The unit test proves the transaction boundary. The injection proves the pool.
• Support reads the new macro in the same pull request as the status field.

Expected result

The ADR cites option C, records option D as refused until catalog accepts a staleness number, and records the dissent of anyone who wanted a broker. The implementation links the ADR. The success test is a dashboard: enroll p99, outbox age, send failures. When those traces stay inside the test for an enrollment week, the decision is confirmed. If outbox age exceeds the promise, you do not quietly move the mail call back into the transaction. You supersede the record with the measurement attached.

Enterprise Real-World Scenarios

The broker that was almost the incident response

In the review after the vendor slowdown, the first slide had six new components and a slogan about eventual consistency. The facilitator — acting as the staff owner — refused to discuss component boxes until the frame was on the whiteboard. Writing the success test took ten minutes and deleted half the slide. The test did not require catalog to be asynchronous. It required the vendor's latency to leave the pool.

Security then asked what the event contained. The draft payload included the learner's email, full name, and the lesson title "so the template is easy." That question would have been late if the diagram had already been approved. It was early because the RFC template forced a data-class line. The room dropped the payload to identifiers. The notifications worker, which already has a audited read API for receipt composition, kept the personal data inside a system designed for it.

Two dissenters wanted a broker because the platform roadmap showed one. The owner overruled them in writing: the strategy allows a broker when a consumer is outside the process and the outbox cannot meet the lag promise, and neither condition held. The dissent is in ADR-214. A quarter later the platform team did introduce a broker for a different flow, cross-region cache invalidation, where the consumer set was genuinely unknown. They did not have to unwind a premature enrollment topic. The overruled dissent had kept the option alive without letting it capture this decision.

The operational surprise was social. Support had been telling learners "if you saw success, the email is in your inbox; check spam." That sentence became false on the deploy. The owner treated the macro change as a release blocker, equal to the migration. The first hour of the new path produced thirty tickets that quoted the old macro and then went quiet once the macro matched the architecture. The incident taught the staff group a rule they wrote into the RFC template: if the user-visible promise changes, the people who speak to users are a release dependency, not an audience for the announcement afterward.

A one-way door hidden in an identifier

A second decision, the same quarter, looked smaller. Enterprise tenants wanted enrollments in a nightly export. An engineer added tenant_key to the outbox payload and to the enrollment primary key "while we are here," using the tenant's display slug. Design a Multi-Tenant SaaS Platform had already warned that slugs change when sales renames an account. Nobody opened an RFC because the column was one line.

Six weeks later a tenant rebranded. The slug changed. The export had been loaded into a warehouse that used the slug as a foreign key, and a partner API had stored it. There was no undo that fit inside a sprint. The repair was a new immutable tenant_id, a compatibility view, and a quarter of dual writes. The staff review that should have happened at the column took a day. The migration took a quarter.

The framing failure is clear in hindsight and writable in advance. Any identifier that leaves the service is a one-way decision, even when the code change is tiny. The guiding policy added to the strategy note was: outward identifiers are opaque and immutable; renames are attributes. The reopen condition on ADR-214 was also amended: a second consumer of the outbox, including a warehouse, reopens the record before the consumer ships. The warehouse was that consumer. It had simply skipped the door.

The episode is the one to remember in interviews. Reversibility is not a property of how hard the code was to type. It is a property of who has already stored the bytes.

Best Practices

• Write the question before the mechanism, and include the user-visible fact the question is about.
• Put the status quo and the smallest effective change in the option list every time. Make the structural change beat them on the success test.
• Pair every improvement with the response measure it worsens. If you cannot name the worsened measure, you have not finished the option.
• Match ceremony to reversibility. Flags and dates for type 2 choices. A full RFC, dissent, and a written undo path for type 1 choices.
• Keep the RFC as a proposal and the ADR as the decision. Link them. Do not leave the decision only in the comment thread.
• Name one owner. Give vetoes only to people who protect a fact the option can break, and write the fact next to the veto.
• Record overruled dissent in the consequences. Future reopen conditions usually come from that paragraph.
• Cite the standing strategy in the RFC, or say which sentence of the strategy you are changing.
• Change support wording, dashboards, and alarms in the same change as the architecture. A new lag is a new page, with a human.
• Leave the meeting with the decision typed. A nod is not an ADR.
• When a column, identifier, or event will be stored by someone else, treat the change as one-way even if the diff is small.
• Park adjacent ideas on a list with owners. Do not let them rewrite today's question.

Common Mistakes and Anti-Patterns

The mechanism frame. The question is "Kafka or gRPC?" It occurs because the industry conversation is about tools, and the incident review is emotional. The consequence is a platform purchase that does not fix the pool, or a rewrite of a call that had to stay synchronous. The repair is the five-part frame, starting from the user-visible fact.

The single-option RFC. The document describes the author's build and ends with "thoughts?" It occurs because the author has already coded the happy path and wants cover. The consequence is a review that can only nitpick, plus a status quo that was never honestly priced. The repair is to require option A and option B before the meeting is scheduled. A staff engineer sends the RFC back rather than improving it in the room.

False reversibility. "We can change the schema later" accompanies an identifier that partners will store. It occurs because reversibility is judged by the author's repository, not by the copies. The consequence is a migration that costs more than the original feature. The repair is the question: who will persist this byte outside our deploy? If the answer is anyone, write the undo path before merging.

Consensus as unanimity. The decision waits until every commenter is happy, including people with no fact to protect. It occurs because conflict feels like failure. The consequence is either a stalled incident fix or a watered-down design that meets no success test. The repair is RFC 7282's test, translated: unresolved objections that break the test or a veto, not a vote. Overrule taste in writing.

Strategy by slideshow. A quarterly deck says "event-driven enterprise" while ADRs say "outbox, modular monolith." It occurs because strategy is assigned to a planning meeting and records are written by teams. The consequence is new hires implementing the deck. The repair is one strategy note whose sentences are cited by ADRs, with exceptions counted.

Consequences with no obligations. The ADR says the decision "increases agility." It occurs because the template has a section and the author is tired. The consequence is a record that cannot fail a pull request. The repair is lesson 10's bar: the cost, the rejected option, the reopen condition, and the human-visible promise.

Deciding without the pager. Notifications discovers the outbox on Saturday. It occurs because the review invited managers and not the on-call rotation. The consequence is an unowned lag. The repair is a veto for the team that will carry the alarm, asked a specific question inside the time box.

Troubleshooting and Design Review

Symptom: The review spends an hour comparing broker products, and the enroll timeout is unchanged.

Likely cause: the frame is a tool choice. The success test is missing or does not mention the user.

Investigation: ask the owner to read the question in one sentence. If they cannot, stop the product comparison.

Corrective action: rewrite the frame around POST /enrollments and the promise. Reschedule. Do not "pick a broker for now."

Symptom: The ADR and the RFC contradict each other a month later.

Likely cause: the RFC kept evolving after the decision, or the ADR was copied from an early draft.

Investigation: compare status headers and dates. Find the pull request that implemented the flow and see which document it links.

Corrective action: freeze the RFC as Decided — see ADR-n. Edit only the ADR's status through the superseding process in lesson 10.

Symptom: Two teams both believe they own the decision, and the work is half done in two styles.

Likely cause: the owner line lists a committee.

Investigation: read the RFC header. If it says "platform and enrollment," that is the defect.

Corrective action: a director or the staff engineer names one owner in writing. The other party keeps a veto only if they still protect a fact. Re-cut scope so one owner can finish.

Symptom: Option C shipped, and support tickets quote a promise the system no longer keeps.

Likely cause: the user-facing sentence was treated as documentation rather than as part of the change.

Investigation: read the macro and the 201 body side by side. Check whether the release checklist includes support.

Corrective action: change the macro the same day. If product never agreed to the weaker promise, the decision was not actually made; stop the rollout and put product's veto on the record.

Symptom: A "temporary" projection of catalog state is now the source enroll uses during incidents, and catalog publishes are ignored for hours.

Likely cause: a type 2 cache became a second writer without a staleness veto.

Investigation: measure the age of the projection at decision time. Find the fallback code path that ignores catalog errors.

Corrective action: fail closed until a new ADR records an accepted age, a rebuild, and catalog's agreement. Delete the silent fallback.

Symptom: Every small change waits three weeks for an architecture board.

Likely cause: type 2 decisions were given type 1 ceremony, usually after a one-way mistake made the organization fearful.

Investigation: sample five tickets that went to the board. Mark which ones were reversible.

Corrective action: publish the reversibility rule. Timeouts, bulkheads, and internal refactors inside one team proceed with a short note. Identifiers, public contracts, and new operational dependencies still take an RFC.

Design-review questions for this lesson: What is the question, in one sentence, and what will it not settle? Who is the owner, and which vetoes protect which facts? What is the success test, with a number and a place it is measured? Is the status quo priced? Which option is the smallest change? What becomes worse if the preferred option wins? Is this reversible, and who already stores the bytes? Where is dissent written? Which strategy sentence does this follow or amend? What will cause us to reopen?

Interview Questions and Answers

Q1. What is the difference between an engineering RFC and an ADR in your process?

An RFC is a proposal opened so people can object while a change is still cheap. It carries a frame, several options, and a preference that is labeled as a preference. An ADR is the owner's decision after that comment window. It records the option accepted, the option refused, the obligations, and the reopen condition. A merged RFC is not a decision, and a decision that exists only in chat is not an ADR. I link them and freeze the RFC once the ADR is accepted. Lesson 10 is the record template. This distinction is the workflow around it.

Q2. How do you frame a decision so the room does not jump to a tool?

I write five lines: the question, the single owner, the time box, the success test with a measure, and the non-goals. The question names a user-visible fact, such as whether enroll stays correct when mail is slow. "Should we use Kafka?" is not a frame. I also write the diagnosis in one or two sentences, because a guiding policy without a diagnosis is a slogan. If the incident was pool exhaustion, the policy is about which calls may hold a transaction, not about a platform standard.

Q3. Why is the status quo a mandatory option?

Because the alternative has to beat something real. Teams omit the status quo when they have already built the new path and want the review to bless it. Pricing the status quo — failure rate, latency, operational pain — sometimes shows that a timeout is enough, and sometimes shows that the pain is large enough to justify a structural change. Either result is a decision. A review that cannot explain why "do nothing" lost has not finished.

Q4. How do you tell a reversible decision from a one-way door?

I ask who will store the outcome outside the committing transaction: another team, a warehouse, a partner, a backup, a public client. I ask whether an identifier's meaning can change, and whether a new pager rotation appears. Code that reverts cleanly can still be one-way if a slug has escaped into someone else's foreign key. Reversible choices get a flag, a metric, and a delete-by date. One-way choices get an RFC, written dissent, and an undo path authored before the bytes escape. The ceremony follows that classification, not the size of the diff.

Q5. A senior engineer wants to replace every synchronous call with events after an outage. How do you respond?

I separate the calls by whether the HTTP response needs their answer. In the enrollment case, the email vendor does not belong inside the transaction; an outbox preserves "we took responsibility to send" without holding a pool connection. The catalog published check does belong on the path if the response must deny an unpublished lesson and catalog has not accepted staleness. Replacing both with one mechanism would fix the outage and break the product sentence. I would also ask which new failure the event path adds: lag, duplicates, and an unowned consumer. Those belong in the option, or the option is incomplete.

Q6. What does rough consensus mean in an architecture review?

It means the owner can name the remaining objections and say which were incorporated, which were overruled and why, and which are unresolved vetoes. It does not mean everyone votes yes, and it does not mean the loudest dissent wins. I take that test from the IETF's description of rough consensus, not from a corporate slogan. Taste is recordable dissent. A veto is a fact the objector protects, such as an on-call alarm with no owner or a freshness promise catalog will not make. If a veto is unresolved, I narrow the decision until it no longer depends on that fact.

Q7. What is a technical strategy, as opposed to a single ADR?

A strategy is a standing diagnosis, a guiding policy, and a short list of refusals that the next ADRs are expected to follow. "Stay a modular monolith until a compliance boundary or a separate on-call team forces a split" is a strategy sentence. An ADR is one choice under that sentence, or a deliberate amendment of it. I keep the strategy alive by citing it in each record and by counting exceptions. A deck that says "event-driven" while the records say "outbox" is not a strategy. It is how new hires build the wrong system.

Q8. How do you handle a stakeholder who was not consulted and blocks the rollout?

I treat it as a framing defect if they protect a fact the option changes: support's promise, security's data class, the pager. The rollout stops if the promise users hear is false, because shipping a known lie is not bias for action. Then I put them on the map with a specific question and a veto limited to that fact. I do not give them ownership of the whole decision as an apology. If they had no fact to protect and simply dislike the mechanism, the owner can proceed and write the dissent down. The repair for next time is the stakeholder table inside the RFC, filled in before the meeting.

Q9. What do you put in the consequences section so it governs later pull requests?

Obligations a reviewer can point at. The promise changed from "mail accepted" to "mail owed within fifteen minutes." The dashboard for outbox age is part of done. Personal data is not in the payload. A second consumer reopens the record. Option D stays refused until catalog names a staleness window. I avoid abstract benefits. A pull request that adds a broker client to enroll fails because the consequence said we do not, not because I remember the meeting.

Q10. How would you run the decision meeting itself?

The RFC is pre-read. I do not present it. We discuss objections that survived the comment window, classified as incorporated, overruled, or veto. Adjacent ideas go to a list with owners so they cannot widen the question. The author has to explain the status quo fairly before arguing for a preference. In the last five minutes the owner reads the decision, the rejection, and the reopen condition, and someone types the ADR before we leave. If we cannot decide, the missing fact and the next meeting are scheduled in the room. An interesting conversation with no sentence at the end is a failed review.

Q11. When would you actually choose the full event-driven option?

When the success test cannot be met by a local outbox, and the veto holders have accepted the consistency loss. Typical triggers: more than one team must react to the enrollment, the worker needs to run in another region, or catalog's availability is the incident and product has signed a staleness number with a rebuild plan. Even then I would keep the event contract small, name the consumer owner, and define duplicates. I would not choose it because a roadmap picture already has a bus. The picture is not a driver.

Hands-On Exercises

Exercise: RFC for replacing synchronous calls with event-driven integration

Problem statement. TechLearningPro's partner webhook today is invoked inside the order transaction: when an order is placed, the API calls the partner's HTTPS endpoint and commits only if the partner returns 2xx. Partners time out during their deploys, and checkout p99 tracks their latency. Write an RFC and the ADR you would accept. You may conclude that a fully asynchronous design is wrong for part of the path. That conclusion is a valid answer if the frame supports it.

Tasks.

1. Write the five-part frame. State the non-goals, including whether payment capture is in this decision. It should not be; capture has its own consistency needs in Design a Payment Processing System.
2. Price the status quo with assumed numbers you label. Include at least the smallest change and one structural change.
3. Classify each option as reversible or one-way, and say who would store the payload.
4. Draw the stakeholder map. Give security a concrete data-class question and on-call a concrete lag question.
5. Draft the ADR consequences as obligations, including support wording and the reopen condition.
6. Specify the failure note for "partner is down after we have committed the order." Say what the learner and the partner observe.
7. Name the standing strategy sentence you are following, from the modular-monolith record, or the sentence you are amending.

Expected deliverables. A two-to-four-page RFC, a one-page ADR, one sequence diagram of the option you accept, and the stakeholder table.

Evaluation rubric.

An RFC that contains only the preferred architecture, or an ADR whose consequence is "better decoupling," is not ready to review.

Lesson Summary

Staff decision-making is a frame, a fair set of options, a reversibility check, a proposal distinct from the decision, and a room that ends with obligations. The enrollment outage looked like a mandate to become event-driven. The frame showed two calls with different jobs. Mail can leave the transaction through an outbox the codebase already understands, with an honest PENDING receipt and a lag alarm someone owns. Catalog's published fact stays a synchronous, fail-closed read until the owner of that fact accepts staleness. The broker remains available for a decision whose success test actually needs it.

The social machinery is part of the design. One owner, vetoes tied to facts, dissent written down, strategy sentences cited, support language shipped with the code. Type 1 and type 2 choices get different ceremony because the cost of being wrong is different, and that cost is paid by whoever has already stored the bytes, not by the size of the diff.

Non-Functional Requirements and SLO Design takes the success test from this lesson and turns it into SLIs, SLOs, and an error budget the next decision is allowed to spend.

Further Reading

• Michael Nygard, "Documenting Architecture Decisions" — the short ADR form lesson 10 adopted. This lesson adds the proposal and the meeting that precede the file.
• IETF RFC 7282, "On Consensus and Humming in the IETF" — rough consensus as unresolved objections, which is the review test used here.
• Jeff Bezos, 2015 letter to shareholders — type 1 and type 2 decisions. Use the distinction to set ceremony, not as a slogan for skipping review.
• Richard Rumelt, Good Strategy/Bad Strategy — diagnosis, guiding policy, and coherent action, scaled here to a technical strategy note.
• Will Larson, Staff Engineer — staff scope when no single team owns the outcome.
• Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns — the coupling you still have after you remove a synchronous call. Lesson 47 applies the catalog.
• Bass, Clements, and Kazman, Software Architecture in Practice — trade-off points, so every option names the measure it worsens.
• This course: Architecture Documentation and ADRs, Event-Driven Architecture Fundamentals, High-Level System Design Methodology, Message Queues and Event Streaming, Architecture Migration and Modernization, Idempotency and Exactly-Once Processing.