Design a Notification Platform

Advanced System Design & Software Architecture · Lesson 56 · Real-World System Design Case Studies

Lesson Overview

A certificate launch emails two million learners. In the same minute, a learner in another region requests a one-time passcode. The announcement workers have filled every thread, the SMS provider is returning 429, and the passcode is still in a queue behind a marketing campaign. The learner cannot sign in. The announcement, which could have waited fifteen minutes, has taken down the only message that could not wait.

TechLearningPro Notify is the platform that sends email, SMS, and mobile push for the learning product. It is not the chat system from Design a Messaging System. Chat is a conversation between people. Notify is the product talking to a person: a passcode, a receipt, a grade, a digest, a course announcement. The difference drives the design. A chat message is stored for both parties and delivered over a socket. A notification is a rendered template, checked against consent, fanned out to one or more channels, handed to a provider, and then tracked until we know what "sent" actually meant.

This lesson designs that platform: a small synchronous API, a durable campaign for broadcasts, per-channel worker pools so marketing cannot block a passcode, preference and suppression checks at send time, idempotency so a retry does not mint a second passcode, rate limits at the recipient, the tenant, and the provider, and honest delivery receipts. Provider calls use documented protocols. Email leaves us over SMTP as specified in RFC 5321, or over a relay's documented submit API, with the host in configuration. Apple Push Notification service is the HTTP/2 interface Apple publishes (api.push.apple.com, device token in the path, apns-id as our idempotency UUID). Firebase Cloud Messaging is the HTTP v1 messages:send method Google publishes. SMS uses whichever provider's documented messages resource is configured; a status callback is registered only when that provider documents one. This lesson does not invent a vendor SDK.

The same idempotency habit as Idempotency and Exactly-Once Processing applies, with a softer correctness bar than payments. A duplicate push is annoying. A duplicate charge is a defect. A duplicate passcode with a new secret is a defect, so the secret is generated once and retries resend the same payload. Queues are at least once, as Message Queues and Event Streaming established. Rate Limiting, Throttling and Load Shedding and Circuit Breaker, Retry and Bulkhead Patterns are how the passcode pool stays alive when the announcement does not.

Module 6 will ask you to defend the SLOs by class of message, the threat of a template that leaks another tenant's data, and the capacity arithmetic of a fan-out. System Design Interview Framework treats "design notifications" as a fan-out and priority problem, not as a single sendEmail box.

Learning Objectives

By the end of this lesson, you should be able to:

1. Separate transactional, security, and marketing notifications, and say which preferences each one must honor.
2. Write functional and non-functional requirements that give a passcode a different latency and isolation budget from a broadcast.
3. Estimate a two-million-recipient campaign in sends per second, queue depth, and storage, and show why the HTTP request must not loop the audience.
4. Specify the API for a single send and for a campaign, including idempotency keys and per-delivery uniqueness.
5. Draw the pipeline from accept, through preference and suppression, to per-channel workers and provider adapters.
6. Explain what a delivery receipt means on email, SMS, and push, and which states we are allowed to call "delivered."
7. Place rate limits on the recipient, the tenant, and the provider account, using separate worker pools as the bulkhead.
8. Handle provider throttling, invalid device tokens, bounces, and a crash between provider acceptance and our commit.
9. Keep tenant and template data from leaking across recipients, and authenticate provider webhooks.
10. Compare per-recipient fan-out with a provider topic broadcast, and say when each is acceptable.
11. Review a design that shares one queue across passcodes and marketing.
12. Carry the case into a Module 6 interview: SLOs by class, failure isolation, and an explicit at-least-once story.

Prerequisites

• Lesson 33: at-least-once delivery, consumer lag, and dead-letter queues. Notify's workers are those consumers.
• Lesson 35: token buckets and load shedding. Provider 429s are a rate-limit problem, not a retry storm.
• Lesson 37: an idempotency key scoped to the caller, stored with a request hash. Here the costly duplicate is a second passcode or a second campaign, not a second journal.
• Lesson 41: bulkheads. The passcode pool and the marketing pool do not share a concurrency limit.
• Lesson 43: metrics that must not explode cardinality. template_id and channel are safe labels. recipient_id is not.
• Lesson 50, if you have read it: user-to-user messaging. Do not import its presence and conversation model into this design. A notification has no reply thread.
• Lesson 55's separation of checkout traffic from webhook traffic. Notify copies that idea: send API, provider webhooks, and campaign workers are three deployments.

Detailed Core Concepts

What a notification is

A notification request is one decision to inform one or more people about one fact, using a template and a priority. A delivery is one attempt to one recipient on one channel. A campaign to two million learners who each have email and a phone is one request and up to four million deliveries. Collapsing those into one row makes receipts, retries, and preferences impossible to express.

Priority classes are part of the data model, not a comment in the worker:

Security messages ignore quiet hours. Marketing messages do not. That single rule prevents the "we waited until morning to send the passcode" incident and the "we sent a promotion at 03:00 local" incident.

Functional requirements

Non-functional requirements

Out of scope: an in-app inbox with read cursors (that is closer to messaging), a full marketing-automation journey builder, and inbound two-way SMS conversations. A STOP keyword that suppresses further SMS is in scope. A chatbot is not.

Preferences, consent, and suppression

Preferences are a matrix of category and channel for a principal. Consent for marketing is stored with a timestamp and a source (settings page, import, unsubscribe link). The send worker reads this at the moment of send. A learner who unsubscribes while a two-million campaign is in flight must not receive the remaining channel copy. Checking only when the campaign is created is simpler and fails that expectation.

Suppression is stronger than a preference. A hard bounce, a complaint, or an SMS STOP writes a suppression row keyed by channel address. Preferences say "this person does not want marketing email." Suppression says "this address does not work or has legally opted out of the channel." Security email to a hard-bounced address still does not send; we fall through to another verified channel and surface SUPPRESSED on the bounced one. If every channel is suppressed, the request completes as undeliverable and the account service, not Notify, decides how the learner recovers.

Quiet hours store a timezone and a window. Timezone comes from the profile, not from the server's zone. A missing timezone skips quiet hours for marketing and logs a metric, rather than assuming UTC and waking a learner in another region. The alternative, dropping the send, hides product announcements. Pick the skip-and-measure behavior and document it.

Unsubscribe links are signed tokens that identify the recipient and the category, with an expiry measured in months, not a guessable account id in the query string. RFC 8058 one-click List-Unsubscribe is added on marketing email when the relay documents support. The unsubscribe endpoint is anonymous, rate limited, and idempotent.

Fan-out

The accept handler for a broadcast writes one campaigns row and one outbox or queue message CampaignAccepted. It does not query two million followers. A fan-out worker reads the audience in pages of a few thousand, ordered by recipient id, and inserts delivery rows in batches with ON CONFLICT DO NOTHING on the unique key. The page cursor is stored on the campaign so a crashed worker resumes mid-audience. Inserting deliveries is the expensive step. Sending is a later step performed by channel workers that select QUEUED rows.

This is fan-out on write of delivery records, which is correct for notifications because each recipient's preference and receipt differ. It is not the social-feed fan-out of lesson 57, where the same post id is copied into many timelines. Do not mix the two designs in an interview. The celebrity problem does not apply. The audience-size problem does.

Provider topic features, such as FCM topics documented by Firebase, can address a large set with one HTTP call. They fit a marketing blast that does not need a per-device receipt and can tolerate a coarser preference model (devices subscribe to a topic only if the user opted in, and unsubscribe removes the device from the topic). They fit a passcode poorly: a passcode is one person, one secret, one receipt. This design uses per-delivery sends for security and transactional traffic. Marketing may use a topic only when product accepts weaker receipts. The default in this lesson is per-delivery, because TechLearningPro's announcements are part of a learner's record ("did we tell them the exam moved?").

What "delivered" means

Honesty about receipts is the difference between a platform and a log line that says "sent."

Callers that need a guarantee, such as "the learner saw the exam change," cannot take it from push. They need an in-product acknowledgement, which is outside Notify. Notify's contract is: we attempted the channels you asked for, subject to preference and suppression, and here is how far each provider confirmed.

Idempotency and the passcode

The caller supplies Idempotency-Key. For a passcode, the account service generates the secret, stores its hash, and calls Notify with the secret in the variable map and a key otp:{challengeId}. A retry of that HTTP call returns the original request and does not enqueue a second delivery set. The worker that crashes after the provider accepts and before we mark SENT_TO_PROVIDER may send twice. Both sends carry the same secret because the payload was fixed on the delivery row before the first call. The learner sees two copies of one code. That is acceptable. Generating a new code on the retry is not: the first code may already be in their inbox and the hash in the account service would no longer match if someone had overwritten it. The account service, not Notify, owns the hash. Notify stores the rendered body encrypted, with a short retention, so a retry can resend without asking the account service to invent a new secret.

Campaign idempotency uses the same shape. A double-clicked "announce" from an admin tool must not create two campaigns. The unique key is (tenant_id, idempotency_key).

Architecture and Design Diagrams

Accept-path traffic, send workers, and provider webhooks scale independently. A webhook replay after an outage must not occupy the threads that accept passcodes.

mermaidCopy
flowchart LR
  subgraph callers [Callers]
    Account[Account service]
    Course[Course service]
    Admin[Admin console]
  end

  subgraph edge [Edge]
    ApiLB[API load balancer]
    HookLB[Webhook load balancer]
  end

  subgraph notify [Notify]
    API[Notification API]
    DB[(PostgreSQL)]
    Cache[(Redis preferences and limits)]
    Fan[Fan-out workers]
    SecQ[Security queue]
    TxQ[Transactional queue]
    MktQ[Marketing queue]
    SecW[Security workers]
    TxW[Transactional workers]
    MktW[Marketing workers]
  end

  subgraph providers [Providers]
    SMTP[SMTP relay]
    APNs[APNs]
    FCM[FCM]
    SMS[SMS provider]
  end

  Account --> ApiLB
  Course --> ApiLB
  Admin --> ApiLB
  ApiLB --> API --> DB
  API --> Cache
  API --> Fan
  Fan --> DB
  Fan --> SecQ
  Fan --> TxQ
  Fan --> MktQ
  SecQ --> SecW
  TxQ --> TxW
  MktQ --> MktW
  SecW --> SMTP
  SecW --> APNs
  SecW --> FCM
  SecW --> SMS
  TxW --> SMTP
  MktW --> SMTP
  HookLB --> API
  SMTP --> HookLB
  SMS --> HookLB

A single security send commits the delivery before any provider I/O. Channel failover happens only from a state that has not been accepted.

mermaidCopy
sequenceDiagram
  participant A as Account service
  participant N as Notification API
  participant DB as PostgreSQL
  participant W as Security worker
  participant P as Channel provider

  A->>N: POST /v1/notifications Idempotency-Key
  N->>DB: Insert request and delivery rows QUEUED
  N-->>A: 202 with delivery ids
  W->>DB: Lease QUEUED row to SENDING
  W->>P: Send fixed payload with provider idempotency id
  alt Accepted
    P-->>W: Acceptance id
    W->>DB: SENT_TO_PROVIDER
  else Throttled
    P-->>W: 429
    W->>DB: Back to QUEUED with not-before
  else Token dead
    P-->>W: Unregistered
    W->>DB: FAILED and disable device token
  end

Component ownership:

The queue technology can be the same broker as lesson 33. The design requirement is three logical queues with independent consumers. One physical topic with a priority header fails the isolation test the moment a slow consumer shares a partition. Use separate topics or separate consumer groups with separate lag SLOs. Security lag pages immediately. Marketing lag pages when the 15-minute campaign SLO is at risk.

Technical Explanation

Capacity

Assumptions:

• 15 million learners with a reachable channel. Peak broadcast: 2 million recipients, email and push, so 4 million deliveries.
• Security traffic: 20 passcodes per second at peak (launch morning), each one channel on average, sometimes two.
• Transactional: receipts and grades, 50 per second peak.
• Audience page size 2,000 recipients. Delivery row about 500 bytes plus a short encrypted payload for security messages.
• Provider budgets we must confirm in their current documentation and in our contract: plan push workers for 5,000 accepts per second aggregate, email for 1,000 per second, SMS for far less and treated as a scarce channel.

Arithmetic:

• Broadcast insert rate. If fan-out runs at 20,000 delivery inserts per second, 4 million rows take 200 seconds. That fits inside the 15-minute SLO with room for the send phase.
• Push send at 5,000 per second clears 2 million pushes in about 400 seconds. Email at 1,000 per second clears 2 million in about 2,000 seconds, which is 33 minutes and misses a 15-minute email SLO. Say this out loud. Either the email SLO applies to "accepted onto the provider" with a higher contracted rate, or the product SLO is "push within 15 minutes and email within an hour," or we raise email parallelism until the relay's limit. The honest design picks a number the relay confirms. A diagram with a cloud labeled "email" is not a capacity plan.
• Security at 20 per second is a rounding error next to the broadcast, and it is the traffic that must not wait behind those 4 million rows. Isolation is a correctness requirement wearing a capacity costume.
• Storage for one campaign's delivery rows: 4e6 * 500 bytes ≈ 2 GB. Rendered marketing bodies are not stored; the template id and variable hash are enough to explain what was sent. Security payloads are encrypted and deleted after seven days. A year of transactional deliveries at 50 per second average is 50 * 86400 * 365 ≈ 1.6 billion rows if the peak were the average, which it is not. Use the real average. At 5 per second average, a year is about 160 million rows, roughly 80 GB before indexes. Partition deliveries by month. This is a large table and a normal one.
• Queue depth: a 4 million message backlog at a few hundred bytes of pointer (delivery id, not body) is small. The body stays in PostgreSQL. Do not put rendered email in the queue payload or you will pay for it twice and retain it in the broker.

The tier that saturates first is the email relay's accept rate, then delivery-row insert rate on the primary, then push. SMS saturates on contract almost immediately and must not be the default channel for a 2 million campaign. Product chooses push plus email for announcements. SMS is for security fallback and explicit opt-in alerts.

APIs we own

POST /v1/notifications with Idempotency-Key.

jsonCopy
{
  "priority": "security",
  "template_id": "otp_login",
  "locale": "en",
  "variables": { "code": "482913", "minutes": "10" },
  "recipient_id": "usr_19",
  "channels": ["sms", "email"]
}

The server checks that the caller may notify usr_19 in this tenant. A course service credential cannot address an arbitrary phone number; it addresses a principal id, and Notify looks up addresses. Letting callers pass raw emails for transactional sends is how one tenant emails another's customers. Raw addresses are allowed only for a dedicated verified-sender flow (adding an email to your own account) and are still rate limited.

Response 202 with request_id and delivery ids in QUEUED. Replay returns that body. The passcode's caller does not need to wait for the provider. If product insists on a synchronous result for security, the API may wait up to two seconds and then return the current state, but the delivery row already exists so a client retry does not create another one.

POST /v1/campaigns with template, audience id, priority marketing or transactional, and an idempotency key. Response 202 with campaign_id. GET /v1/campaigns/{id} returns counts by state, which is the admin progress bar. Counts come from a periodic aggregate, not from COUNT(*) on a hot partition in the request path.

GET /v1/deliveries/{id} for the calling service. Learners see their own recent notifications through the BFF, which filters by principal.

POST /v1/provider-webhooks/{providerCode} verifies the provider's documented signature on the raw body, dedupes on the provider event id, and updates the delivery or writes a suppression. Unknown event types are stored and acknowledged so a new event type does not retry forever, and a metric marks them for a parser update.

POST /v1/unsubscribe accepts the signed token and writes the preference. It is idempotent.

Rate limits return 429 with Retry-After. A limit hit on marketing does not mark the delivery FAILED permanently; it sets not_before. A limit hit because the recipient has had enough marketing this week marks that delivery SUPPRESSED with reason frequency_cap and does not retry it tomorrow unless the request was transactional.

Schema

sqlCopy
CREATE TABLE notification_requests (
  request_id        uuid PRIMARY KEY,
  tenant_id         uuid NOT NULL,
  idempotency_key   text NOT NULL,
  request_hash      bytea NOT NULL,
  priority          text NOT NULL,
  template_id       text NOT NULL,
  locale            text NOT NULL,
  recipient_id      uuid,
  campaign_id       uuid,
  created_at        timestamptz NOT NULL,
  UNIQUE (tenant_id, idempotency_key)
);

CREATE TABLE campaigns (
  campaign_id     uuid PRIMARY KEY,
  tenant_id       uuid NOT NULL,
  audience_id     uuid NOT NULL,
  priority        text NOT NULL,
  template_id     text NOT NULL,
  cursor_recipient uuid,
  state           text NOT NULL,
  created_at      timestamptz NOT NULL
);

CREATE TABLE deliveries (
  delivery_id     uuid PRIMARY KEY,
  tenant_id       uuid NOT NULL,
  request_id      uuid,
  campaign_id     uuid,
  recipient_id    uuid NOT NULL,
  channel         text NOT NULL,
  address_ref     text NOT NULL,
  state           text NOT NULL,
  provider_code   text,
  provider_id     text,
  not_before      timestamptz,
  attempt_count   int NOT NULL DEFAULT 0,
  created_at      timestamptz NOT NULL,
  updated_at      timestamptz NOT NULL,
  UNIQUE (campaign_id, recipient_id, channel),
  UNIQUE (request_id, recipient_id, channel)
);

CREATE TABLE preferences (
  tenant_id     uuid NOT NULL,
  principal_id  uuid NOT NULL,
  category      text NOT NULL,
  channel       text NOT NULL,
  enabled       boolean NOT NULL,
  updated_at    timestamptz NOT NULL,
  PRIMARY KEY (tenant_id, principal_id, category, channel)
);

CREATE TABLE suppressions (
  tenant_id    uuid NOT NULL,
  channel      text NOT NULL,
  address_hash bytea NOT NULL,
  reason       text NOT NULL,
  created_at   timestamptz NOT NULL,
  PRIMARY KEY (tenant_id, channel, address_hash)
);

CREATE TABLE device_tokens (
  token_hash    bytea PRIMARY KEY,
  tenant_id     uuid NOT NULL,
  principal_id  uuid NOT NULL,
  provider      text NOT NULL,
  disabled_at   timestamptz
);

address_ref points at the account service's address record. Notify does not become the system of record for every email in the company, though it does store a hash for suppression so a bounce can be enforced without a join on the hot path. The unique constraints on deliveries are the fan-out's idempotency. Partial uniqueness needs care in PostgreSQL: a campaign delivery has a null request_id and a single-recipient request has a null campaign_id. Use two partial unique indexes instead of a constraint that treats all-null keys as distinct. The migration uses:

sqlCopy
CREATE UNIQUE INDEX deliveries_campaign_once
  ON deliveries (campaign_id, recipient_id, channel)
  WHERE campaign_id IS NOT NULL;

CREATE UNIQUE INDEX deliveries_request_once
  ON deliveries (request_id, recipient_id, channel)
  WHERE request_id IS NOT NULL;

Templates live in templates (tenant_id, template_id, locale, channel, version, body) with body in a logic-less format. Rendering escapes HTML in email bodies. There is no expression language and no "template from the request." A caller-supplied body would be an injection and a compliance bypass.

Scale, cache, and load balancing

The API is stateless behind an L7 load balancer, round-robin, /ready checking the primary. Webhooks use a second listener and a second pool. Workers are not on that load balancer; they scale on queue lag. Security worker autoscaling tracks security lag with a low threshold. Marketing autoscaling has a ceiling so a campaign cannot consume every database connection. That ceiling is the bulkhead when the queue and the database are shared.

Cache:

Provider clients have their own connection pools and timeouts. APNs wants long-lived HTTP/2 connections; the adapter maintains a small pool per worker process rather than a connection per send. FCM uses the documented OAuth bearer token, cached until shortly before expiry. Secrets and the APNs signing key stay in a secret manager. The device token is a bearer-like identifier: store a hash for the primary key if you can still address the provider, which you cannot, because APNs requires the token itself. Store the token encrypted, and log only the hash.

Read replicas serve campaign progress aggregates. State transitions read and write the primary. A worker that leases a row uses UPDATE deliveries SET state = 'SENDING' WHERE delivery_id = ? AND state = 'QUEUED' RETURNING * so two workers cannot send the same row. The lease expires: if updated_at is older than two minutes and the state is still SENDING, a sweeper returns it to QUEUED. That sweeper is why the payload must be safe to send twice.

Security

Tenant id comes from the service credential or from the host-mapped tenant in lesson 60's sense, not from a body field alone. Every query includes tenant_id. A campaign audience query is constrained to that tenant at the account service.

Templates cannot call out to the network or execute code. Variables are data. HTML email escapes them. A learner's display name is a classic injection string; treat it as one.

Webhook signatures follow the provider's current documentation. Fail closed. Dedupe after the signature check, as in lesson 55: a forged event must not consume the event id.

PII minimization: logs contain delivery_id, template_id, channel, and the provider status code. They do not contain the passcode, the full email address, or the device token. Traces use the same rule. Support tools that show an address require a role and an audit event.

Failover: if the primary SMS provider returns unavailable and the delivery is still SENDING with no acceptance id, a second configured provider may be tried and the row records which provider accepted. If an acceptance id exists, we do not send the same delivery to another provider. Two SMS messages with the same code are tolerable; two providers and no record of the first is how we lose the audit. Email failover follows the same rule. Push failover across APNs and FCM is not failover: they are different devices. Each device is its own delivery.

Failure modes

Trade-offs

Per-delivery fan-out versus provider topics. Per-delivery gives receipts, quiet hours, and a precise unsubscribe. It costs rows and provider calls. Topics cost one call and give up those controls. Use topics only for marketing that product agrees does not need a receipt.

Queue versus send inside the HTTP request. In-request send makes passcode latency look smaller until the provider stalls and the API's threads are gone. The queue adds a hop of tens of milliseconds and preserves isolation. Security workers stay near empty. That hop fits a 5-second p99.

Check preferences at accept and at send. Accept-time checks fail fast for a single recipient ("this user has no SMS"). Send-time checks honor unsubscribes during a long campaign. Pay for both. Accept-time only is a product bug. Send-time only wastes queue space on traffic we already knew was suppressed.

Store rendered bodies versus re-render on retry. Storing the security payload (encrypted) makes retries stable. Re-rendering marketing from the template means a template edit mid-campaign changes the remaining sends. That is usually what an admin wants when they fix a typo, and it is dangerous when they change a link that half the audience already received. Freeze the template version on the campaign at start. Edits create a new version for the next campaign.

Shared database versus a dedicated delivery store. PostgreSQL is enough at the rates above if deliveries are append-mostly and workers update by primary key. A wide-column store becomes attractive when delivery updates dominate and the table is multi-terabyte. Do not start there. The unique constraints are the feature you would be rebuilding.

Practical Implementation or Design Walkthrough

Walkthrough A, passcode. The account service creates challenge ch_44, generates secret 482913, stores only the hash, and calls POST /v1/notifications with key otp:ch_44, priority security, channels SMS then email. Notify inserts the request and two delivery rows in one transaction and returns 202. The security worker leases the SMS row, re-reads suppression from the primary, renders the template, and calls the configured SMS provider with a provider idempotency id equal to delivery_id when that provider documents an idempotency field. On acceptance it stores the provider id and SENT_TO_PROVIDER. A callback later moves it to DELIVERED if the documented status says so. If SMS returns a permanent failure, the worker leaves a FAILED row and the email delivery, already queued, proceeds. The account service does not generate a second secret when Notify retries.

Walkthrough B, announcement. An admin submits a campaign with key announce:cert:2026-11-06 and an audience id for "learners enrolled in the certificate." The API writes the campaign in FANOUT_RUNNING and returns 202. The fan-out worker pages recipient ids, inserts email and push deliveries, and advances cursor_recipient in the same transaction as the page insert. Channel workers render version 3 of the template, the version frozen on the campaign. A learner who unsubscribes at page 400 is inserted if the page was built before the unsubscribe, and skipped at send time when the worker re-checks preferences. Progress counts update every few seconds from GROUP BY state on that campaign's partition, run by a job, not by the admin's browser hitting the primary in a loop.

Walkthrough C, bad token. APNs returns 410. The worker sets the delivery to FAILED, sets device_tokens.disabled_at, and does not open a new delivery to "try email" unless the original request asked for email. Inventing a channel the caller did not ask for surprises learners who never gave that address for this category.

The lease update is ordinary SQL, shown because the race is the design:

javaCopy
public Optional<Delivery> lease(UUID deliveryId) {
  return jdbc.sql("""
      UPDATE deliveries
         SET state = 'SENDING', attempt_count = attempt_count + 1, updated_at = now()
       WHERE delivery_id = ?
         AND state = 'QUEUED'
         AND (not_before IS NULL OR not_before <= now())
      RETURNING delivery_id, channel, state, attempt_count
      """)
      .param(deliveryId)
      .query(Delivery.class)
      .optional();
}

JdbcClient is the Spring API. An empty optional means another worker won or the row is delayed. The sender does nothing in that case. Tests cover: two threads leasing one row, a 429 returning the row to QUEUED, a replayed idempotency key creating zero extra rows, and a marketing worker pool that cannot acquire a security connection budget. The last test is a configuration test on pool sizes, which is where isolation is actually implemented.

Enterprise Real-World Scenarios

Scenario 1: A product launch shares a queue with sign-in

A consumer subscription company sends a "new season" push to tens of millions and, on the same cluster, sends email one-time passcodes for account recovery. One consumer group pulls both. On launch night the passcode p99 goes from two seconds to twenty minutes. Support sees a flood of "I never got the code," and the retry button in the app creates new challenges because the account service treats a client retry as a new secret. Learners now have several codes and type the oldest.

The repair that lasts is structural. Security deliveries move to their own topic and worker deployment with a reserved database pool. The account service uses idempotency key otp:{challengeId} and stops minting a new secret when Notify already has the request. The client retries the same challenge. Metrics page on security lag above a few seconds, not on overall queue depth, which is dominated by marketing and hides the outage. The company also caps marketing send rate below the provider's ceiling so push 429s do not back up into the broker and inflate storage.

They keep per-delivery rows for the season announcement because customer support asks "did this subscriber get the push?" Topic broadcast would have been cheaper and would have made that question unanswerable. They accept the cost. A later digest email, where the only question is "did the job run," moves to a coarser batch report. Different questions, different designs.

Scenario 2: A university tenant and a public MOOC share Notify

TechLearningPro hosts a public course site and a university tenant on one notification platform, the multi-tenant problem lesson 60 treats in general. The university's exam-change email must not be renderable with the MOOC's template, and a compromised course-service credential for the MOOC must not address university students. Tenant id is part of every primary key lookup. Audiences are lists the account service scopes to the tenant. A bug that queries deliveries by recipient id alone is a cross-tenant read; the repository does not offer that method. Webhook routing includes the tenant because provider accounts differ per large tenant: the university brings its own email domain and its own signing secret. Smaller tenants share TechLearningPro's provider account and are isolated by rate limit and by the from-address the adapter is allowed to use.

During finals week the university's transactional volume rivals a marketing blast. Their traffic is transactional, not marketing, so it uses the transactional pool. That pool has a higher ceiling than marketing and a lower one than security. The MOOC's promotional campaign continues on the marketing pool and slows down. Nobody files a ticket that passcodes are late, because the security pool's lag chart stays flat. The design review for Enterprise Architecture Review Checklist asks for those three lag charts. A single "notifications sent" counter would have hidden the finals-week story.

Best Practices

• Model requests, campaigns, and per-channel deliveries as different rows. Receipts live on the delivery.
• Give security, transactional, and marketing separate queues, worker pools, and lag alerts.
• Require an idempotency key. Derive passcode keys from the challenge id. Derive campaign keys from the admin action id.
• Generate a secret once. Retries resend the stored payload.
• Check suppression and marketing consent at send time. Freeze the template version at campaign start.
• Use logic-less templates and HTML-escape variables. Do not accept a caller-supplied body.
• Address principals, not raw emails, except on a verified-address flow.
• Map provider statuses you can cite in their documentation. Leave everything else at SENT_TO_PROVIDER.
• Disable device tokens on the provider's unregistered response. Do not keep sending to a 410.
• Rate limit the recipient, the tenant, and the provider. Shed marketing first. Back off on 429 without burning the attempt as a permanent failure.
• Keep webhook ingress off the accept-path deployment.
• Log ids and status codes. Keep passcodes, device tokens, and full addresses out of logs and traces.
• Page the audience and store the cursor. A campaign worker is resumable.
• Partition deliveries by time. Aggregate campaign counts on a job.
• Prove isolation with a test that marketing backlog does not increase security lag.

Common Mistakes and Anti-Patterns

One queue for every priority. It is fewer moving parts until the day a broadcast delays a passcode. The savings are a topic and a deployment. The incident is a sign-in outage. Separate them from the first design, because retrofitting priority into a single consumer group is how messages get rewritten in production.

A new passcode on every retry. The client times out, the button creates challenge ch_45, and the first SMS arrives with ch_44's code. The hash for ch_44 is the one that will never be typed. Idempotency on the challenge, and a client that reuses it, is the fix. Notify cannot fix a caller that wants a new side effect.

Calling the provider inside the request thread for a broadcast. The admin's HTTP call dies at the load balancer timeout after 30,000 sends, the rest never start, and a retry duplicates the first 30,000. Campaigns are asynchronous. The admin UI polls counts.

Treating provider acceptance as human delivery. Product promises "we will know they saw it" because APNs returned 200. The banner was never shown. Contracts with product use the receipt table in this lesson. In-app acknowledgement is a different feature.

Preference checked once, at enqueue. A campaign that runs for twenty minutes will email people who unsubscribed at minute two. Send-time checks are the product behavior users assume.

Caller-supplied HTML. It lets a tenant brand a message and it lets a compromised caller inject content into another user's mailbox under your domain. Templates are data you review. Variables are escaped.

Logging the OTP to debug a template. The log pipeline becomes a second factor for anyone who can read it. Render in a test with a fixed code. Production logs the delivery id.

Failover after acceptance. Trying a second SMS provider because the callback is slow sends a second message and splits the audit trail. Fail over only when no acceptance id exists.

Unbounded fan-out cardinality in metrics. A label of recipient_id on a counter will take the metrics system down during the campaign. Label by channel, priority, template, and provider status.

Troubleshooting and Design Review

Design-review questions for Module 6:

• Where is the security bulkhead, in concrete pools and topics?
• What is stored before the provider call, and what may be sent twice?
• Which receipt states are claims about the provider, and which would be claims about a human?
• How does unsubscribe win a race with an in-flight campaign?
• What is the email accept rate the SLO assumes, and who signed that contract?
• How does tenant id constrain templates, audiences, and webhooks?
• Which metrics are labeled, and which identifiers are forbidden as labels?

Interview Questions and Answers

Q1. How do you send a passcode and a two-million-person announcement on the same platform?

You give them different priority classes, queues, and worker pools. The passcode is a security request: one or two deliveries, idempotency key derived from the challenge, payload fixed before the first provider call, p99 of a few seconds from enqueue to provider accept. The announcement is a campaign: the HTTP call inserts one campaign row and returns 202, a fan-out worker pages the audience and inserts delivery rows, and marketing workers send under a rate ceiling. They may share PostgreSQL and a broker cluster. They do not share a consumer group or a connection pool. If they do, the announcement's backlog becomes the passcode's latency. Say the numbers: two million pushes at five thousand a second is on the order of seven minutes, and email is usually slower, so the email SLO has to match the relay contract. The passcode's twenty requests a second are irrelevant to throughput and absolute for isolation.

Q2. What does idempotency cover here, and what can still duplicate?

The Idempotency-Key on the request or campaign makes a client retry return the original rows. The unique delivery key (request or campaign, recipient, channel) makes fan-out safe to rerun. The provider is called only after a conditional lease from QUEUED to SENDING. What can still duplicate is the crash after the provider accepts and before we store the acceptance: the sweeper will lease again. Mitigate by sending the same stored payload and, when the provider documents an idempotency id (apns-id, or a field they document for SMS), send the delivery id as that value on every attempt. You do not get exactly-once on a provider that has no such mechanism. You get at-least-once with a stable body. For a passcode that is the right trade. For a payment it would not be, which is why lesson 55's UNCONFIRMED path is stricter.

Q3. When do you check preferences relative to fan-out?

At send time, every time, for the category and channel. Also at accept time for a single-recipient call so you can return a clear "no channel available" without waiting. A campaign's audience snapshot is an eligibility hint, not a permit. People unsubscribe during long sends. Quiet hours are evaluated at send time in the recipient's timezone; if the hour is quiet and the class is marketing, set not_before to the end of the window rather than dropping the message. Security ignores quiet hours. If the preference store is down, marketing stops and security uses the channel list already authorized on the row. Unknown marketing consent is not an implicit opt-in.

Q4. What is the difference between accepted, sent, and delivered?

Accepted means Notify committed a delivery row. Sent to the provider means the provider's API returned success: SMTP 250, APNs success, FCM message name, or the SMS provider's accept code. Delivered means that provider later sent a documented positive receipt, which many email and push paths never do. Product copy and API enums must not collapse these. Support workflows key off bounce and complaint, which are negative receipts, and off SENT_TO_PROVIDER age. A delivery that stays at SENT_TO_PROVIDER is not a failure by itself. A delivery that stays at QUEUED past the class SLO is.

Q5. How do you rate limit without blocking passcodes?

Three buckets: per recipient per category (a person should not get forty marketing pushes in a day), per tenant (contractual quota, and a noisy neighbor on a shared platform), and per provider account (the relay's real ceiling). Buckets are Redis token buckets shared by all workers, otherwise each worker sends the full provider quota. When the provider bucket is empty, marketing workers back off first because their queue is separate and their autoscaler has a ceiling. Security workers have a reserved slice of provider quota if the contract allows, or a separate provider subaccount if it does not. A 429 sets not_before and does not count as a permanent failure until a documented attempt limit. Shedding is 429 to the campaign submitter only when the tenant daily cap is already exhausted at accept time.

Q6. How do device tokens and bounces stay correct?

Push tokens are stored encrypted, keyed for lookup by principal, and disabled when APNs returns 410 or FCM returns its documented unregistered error. Continuing to send wastes quota and can look like abuse to the provider. Email hard bounces and complaints write a suppression on the address hash. The send worker checks suppression before render. SMS STOP, when the provider documents a stop keyword or a status, writes the same kind of row. Soft bounces retry with backoff and do not suppress on the first failure. The account service is notified of a disabled address so the profile UI can ask for a new one; Notify does not delete the user's account.

Q7. Where is the fan-out cursor, and what happens if a worker dies mid-page?

The cursor is campaigns.cursor_recipient, the last recipient id fully inserted. The worker reads the next page, inserts deliveries with ON CONFLICT DO NOTHING, and updates the cursor in one transaction. A crash before commit retries the same page; the unique index makes the retry safe. A crash after commit continues from the cursor. Do not ack a queue message that means "campaign done" until the state is FANOUT_COMPLETE. The channel workers are independent: they can send page 1 while fan-out writes page 50, which is what gets the first recipients their mail before the insert job finishes. If you wait to send until every row exists, you add the full insert time to the first recipient's latency for no consistency benefit.

Q8. How do you isolate tenants on a shared notification platform?

Every row carries tenant_id, and repositories require it. Audiences are produced by a tenant-scoped query. Templates are tenant-owned; a shared global template is an explicit platform template with a fixed id, not a missing tenant predicate. Provider secrets for a large tenant are theirs, and webhook URLs include the provider code and tenant so the signature secret is unambiguous. Shared provider accounts still stamp a from-address and a rate bucket per tenant so one university cannot consume the MOOC's email quota or send as the wrong domain. Cache keys include tenant id. A preference cache without it serves one school's quiet hours to another.

Q9. What do you put in the queue message?

A delivery id and a priority, and little else. The worker loads the row from PostgreSQL, which is the source of truth for state, payload reference, and attempt count. Putting the rendered body in the message duplicates a secret into the broker's retention and makes the queue the hardest store to purge when a passcode must be deleted. It also lets a stale message send a payload the database has already suppressed. Small messages keep the backlog of a four-million campaign cheap. The cost is a primary-key read per send, which is the right trade at these rates.

Q10. How would you fail over email providers?

Only before acceptance. The delivery row records provider_code and provider_id. An empty provider_id and an unavailable error allows one configured backup relay, then the row stores whichever relay accepted. A row that already has a provider id is never sent to the backup, even if the bounce webhook is late; the bounce path suppresses or retries on the same provider according to its status. Health is a circuit breaker per relay from lesson 41, scoped to the marketing and transactional pools. You do not fail a security message over to a marketing-only relay that is not allowed to send from the security domain. Domain alignment and signing (SPF, DKIM, DMARC as operated by the relay) are part of the adapter configuration, not an afterthought in the failover branch.

Q11. A stakeholder wants every notification synchronous and "exactly once." What do you say?

Synchronous send couples the caller's availability to every provider's latency and turns a broadcast into a timeout. You accept synchronously onto a durable row, which is the guarantee the caller can have in milliseconds, and you deliver asynchronously with a class SLO. Exactly-once display on a human's phone is not a property you can get from APNs, FCM, SMTP, or SMS. You can get exactly-once request acceptance via the idempotency key, and at-least-once provider submission with a stable payload and, where documented, a provider idempotency id. Phrase it that way in System Design Interview Framework. If they need the user to have certainly seen the content, add an in-product acknowledgement and stop asking the push provider to prove it.

Q12. Which failures page a human, and which are ordinary?

Page on security lag, on provider 5xx or 429 above a burn rate for the security pool, on campaign fan-out stalled, on suppression-store errors, and on webhook signature failures. Do not page on a single 410, a single hard bounce, or a marketing delivery waiting for quiet hours. Those are metrics and weekly reviews. Do page if the 410 rate collapses a large fraction of tokens after a client release, because the app is sending bad tokens and the campaign will look "sent" in your database and absent on phones. Architecture Performance and Capacity Review is where you show the lag SLO and the relay contract side by side. Disaster Recovery and Business Continuity is where you show that QUEUED rows, not the broker, are the recovery source after a queue loss.

Hands-On Exercises

Exercise 1: Design the launch-day pipeline

Problem. TechLearningPro will announce a new certificate to 2 million learners by push and email, while sign-in passcodes must stay under a 5 second p99 to the provider. SMS is allowed for passcodes only.

Tasks.

1. State functional requirements and the three priority classes.
2. Estimate insert time and send time. Mark which SLO the email relay must agree to.
3. Draw the accept path and the three worker pools.
4. Define the delivery unique key, the campaign cursor, and the idempotency key for a passcode.
5. Specify receipt states for push and email, including one state you refuse to claim.
6. Specify rate limits and what a 429 does to each class.
7. Specify one crash after provider acceptance.
8. List the metrics that page, with labels you allow.

Deliverables. A design note, a sequence diagram, the delivery state list, and a capacity paragraph with arithmetic.

Exercise 2: Review a shared-queue proposal

The proposal: one topic, workers send inline during POST, the client retries passcodes by creating a new code, preferences are copied onto the campaign at start and never read again, and "APNs 200 means delivered." Write the review. Name the isolation mechanism, the idempotency change, the send-time check, and the receipt vocabulary you would require before this ships.

Rubric.

Lesson Summary

TechLearningPro Notify turns one product fact into per-channel deliveries with a priority, a preference check at send time, and a provider accept state that is not exaggerated into "the human saw it." Broadcasts are campaigns with a resumable cursor. Passcodes are small, idempotent, and isolated on their own workers so a two-million-row announcement cannot own the sign-in path. Rate limits sit on the recipient, the tenant, and the provider. Failover happens only before a provider id exists. Templates are versioned and logic-less. Tenant id scopes every read.

The design spends rows and provider calls to keep receipts and consent. Where a blast does not need either, a documented provider topic is the cheaper alternative, and you should say so rather than fan out by habit. Module 6's interview framework, SLO lesson, capacity review, and disaster-recovery lesson are where you defend the class SLOs, the relay contract, and the fact that QUEUED rows are the source of truth when the broker is empty.

Further Reading

• Message Queues and Event Streaming, Rate Limiting, Throttling and Load Shedding, Idempotency and Exactly-Once Processing, and Circuit Breaker, Retry and Bulkhead Patterns.
• Design a Payment Processing System for the stricter form of the same accept-then-call pattern.
• System Design Interview Framework, Non-Functional Requirements and SLO Design, and Architecture Performance and Capacity Review.
• RFC 5321, Simple Mail Transfer Protocol: https://www.rfc-editor.org/rfc/rfc5321.
• RFC 8058, one-click list unsubscribe: https://www.rfc-editor.org/rfc/rfc8058.
• Apple, sending notification requests to APNs: https://developer.apple.com/documentation/usernotifications/sending-notification-requests-to-apns.
• Firebase Cloud Messaging HTTP v1: https://firebase.google.com/docs/cloud-messaging/send/v1-api.
• Twilio's documented Messages resource, as an example of an SMS provider API and status callback, used only when that vendor is the configured adapter: https://www.twilio.com/docs/messaging/api/message-resource.