Design a Payment Processing System

Advanced System Design & Software Architecture · Lesson 55 · Real-World System Design Case Studies

Lesson Overview

A learner buys Distributed Systems in Practice for 49.00 USD. The browser shows a spinner. The phone network drops. The client retries. The acquirer approved the first call. TechLearningPro's API never stored the approval. Finance, three weeks later, has a settlement line the product database cannot explain, and support has a learner who was charged and still sees "payment failed."

That incident is the whole design problem. A payment system is a state machine for money, a ledger that can be audited, a boundary that keeps card data out of our network, and a reconciliation loop that assumes the acquirer and our database will disagree. LLD Case Study: Payment Gateway already fixed the module: states, a PaymentProvider port, a unique (principal_id, idempotency_key), and the rule that a timeout is UNCONFIRMED. Idempotency and Exactly-Once Processing fixed the distributed shape: the key, the request hash, the outbox, the inbox, and the refusal to call a broker flag "exactly-once" when PostgreSQL and the acquirer sit outside it. This lesson is the system around that module.

TechLearningPro Payments is the merchant platform for course purchases, renewals, and refunds. It is not a bank and it is not a cardholder-data store. The browser sends the card to a tokenizer or to provider-hosted fields. Our API receives an opaque payment-method token, an amount in minor units, a currency, an order id, and an Idempotency-Key. We authorize, capture, void, and refund through a provider adapter whose URL comes from configuration and whose contract is that provider's published API reference. We post a balanced journal in the same database transaction as the payment-state change. We accept signed webhooks, dedupe them by the provider's event id, and we reconcile every settlement file against the ledger. A mismatch becomes a case. It does not become a silent update of a learner balance.

The design target in this lesson is a single-merchant learning business that can burst to a few hundred payment attempts per second, with an explicit path to shard by merchant if TechLearningPro later processes payments for a marketplace of instructors. The numbers are worked in the capacity section so a design review can attack the assumptions instead of the adjectives.

Module 6 will grade this case the way a staff review grades it. Non-Functional Requirements and SLO Design asks for the authorize latency and the ledger's recovery point as numbers. Architecture Security and Threat Modeling asks where the PCI boundary is drawn and what a stolen token can do. Disaster Recovery and Business Continuity asks how you promote a replica without double-capturing. System Design Interview Framework uses this lesson as a canonical money workflow. The capstone in Enterprise System Design Capstone Project can extend it with marketplace payouts; this lesson stops at merchant capture, refund, ledger, and reconciliation.

Learning Objectives

By the end of this lesson, you should be able to:

1. State the functional requirements for authorize, capture, void, refund, webhooks, ledger posting, and reconciliation, and name what is out of scope.
2. Turn a checkout workload into attempts per second, row writes, and retention, and say which tier saturates first.
3. Specify the HTTP API we own, including Idempotency-Key, request-hash conflicts, and which operations get their own keys.
4. Draw the payment state machine consistently with lesson 22, including UNCONFIRMED, and refuse illegal transitions.
5. Design a double-entry ledger whose journals balance per currency, and explain why a single balance column is not the book of record.
6. Place the PCI boundary so primary account numbers and card verification values never reach our database, logs, or traces.
7. Describe provider routing as a port plus configuration, including when a second provider is a failover and when it is a double charge.
8. Explain cache, load-balancer, and bulkhead choices that protect checkout from webhook floods and from provider slowness.
9. Walk a crash window: approved at the acquirer, not yet committed locally, and say what resume and reconciliation each do.
10. Compare consistency, latency, and operational cost for synchronous ledger posting versus an eventually consistent ledger.
11. Review a payment design for stuck IN_PROGRESS rows, unsigned webhooks, and settlement breaks.
12. Carry the case into a Module 6 interview or capstone without collapsing it back into a single class diagram.

Prerequisites

• Lesson 22: payment states, PaymentProvider, tokenized methods, idempotency scoped to the principal, provider calls outside the database transaction, webhook signature-then-dedupe. This lesson does not redesign that class model. It places it in a system with a ledger, settlement, and capacity.
• Lesson 37: Idempotency-Key, canonical request hash, IN_PROGRESS versus COMPLETED, 409 on mismatch, outbox written with the state change, inbox on consumers, reconciliation when the provider cannot dedupe a retry. UNCONFIRMED in lesson 22 and UNKNOWN in lesson 37 are the same operational fact: we might have moved money and we have not proved it. This lesson stores that fact as UNCONFIRMED and resolves it by resume or by settlement match.
• Lesson 32 for refunds that cross enrollment and payments with a saga key derived from the refund id, not a fresh UUID per HTTP attempt. Lesson 33 for at-least-once delivery. Lesson 35 for velocity limits on checkout. Lesson 41 for timeouts, retries, and bulkheads around the acquirer. Lesson 44 for service identity between order and payment.
• Comfort with minor-unit money (integer cents, no floating point) and with PostgreSQL unique constraints and transactions. You do not need a merchant account to study the design.

Detailed Core Concepts

What this system is responsible for

A payment processing system, in this course, is the merchant-side orchestration of card and wallet charges for TechLearningPro. The acquirer or payment service provider authorizes against the card network. We decide when to capture, how much we may refund, what our books say, and how we notice when our books and the provider's settlement disagree.

In scope:

• Create a payment for an order, authorize it, and either capture immediately or capture later up to the authorized amount.
• Void an authorization that has not been captured.
• Refund up to the captured amount, in one or more partial refunds.
• Persist an idempotent result for every command the client or a saga may retry.
• Post balanced ledger journals for capture, fee recognition when settlement arrives, and refunds.
• Ingest provider webhooks after signature verification and dedupe them.
• Ingest a settlement file and open a break for every line we cannot match.
• Expose a read model of payment status to the order service and to the learner.

Out of scope, named so an interview does not wander:

• Issuing cards, holding deposits, or extending credit.
• Marketplace payouts to instructors. Those are a later ledger of payables, and they reuse this posting style. They are the capstone extension, not this lesson's critical path.
• Chargeback representment workflows beyond recording a dispute event and freezing the related refundable balance. The dispute desk is a process; the system must not let a refund and a chargeback both pay out the same capture without a human or a documented policy.
• Storing or transmitting a primary account number or a card verification value.

Functional requirements

Non-functional requirements

These numbers are the SLO draft Non-Functional Requirements and SLO Design expects you to defend. If a stakeholder wants end-to-end p99 of 300 ms, the provider call dominates and the requirement is not implementable without a different payment product (for example a stored mandate that is captured asynchronously). Say that in the interview.

Actors and trust boundaries

The learner's browser is untrusted. It may talk to provider-hosted fields. It talks to TechLearningPro's checkout BFF with the learner's session. The BFF is allowed to create a payment for an order the order service says is payable by this principal. The payment service does not trust the browser's amount without the order service: the amount and currency on the payment are copied from the order, and a mismatch is a 409. Otherwise a modified client pays 1 cent for a 49 dollar course.

The payment service is the only component that holds the provider API secret and the only component that writes the ledger. The order service, the enrollment consumer, and the receipt consumer never connect to the provider and never write journals. Webhooks terminate on a dedicated ingress that can see only the webhook authenticator and the payment service, not the checkout BFF.

The provider is outside our trust zone. Their responses are authenticated by TLS to their documented host, and their webhooks are authenticated by the signature scheme their current documentation specifies. We do not invent a signature header. When the configured provider is Stripe, their documented Stripe-Signature scheme (timestamp, payload, HMAC, tolerance window) is what the adapter implements, and the signing secret comes from configuration. Another provider's adapter implements that provider's documented scheme. A missing scheme means we do not accept webhooks from that provider until one exists; polling their documented status query is the alternative.

Payment state machine

The states match lesson 22 so the low-level design and this high-level design describe one system.

CREATED is not a state the client should observe for long. The create path commits IN_PROGRESS and then calls the provider. The client observes AUTHORIZED, DECLINED, or UNCONFIRMED, or a 409 while the first attempt is still in flight.

Illegal transitions are enforced in the payment aggregate before any HTTP call on the way out, and again as a conditional update (UPDATE ... WHERE status = ? AND captured_minor = ?) so two captures cannot both succeed. The database check is the one that survives a bug in a second code path.

Idempotency at system scale

Lesson 37's record is the system rule, repeated here only as the contract every new endpoint must meet:

• Unique (principal_id, idempotency_key).
• SHA-256 of the canonical JSON body: stable key order, amounts as integers, no insignificant whitespace.
• States IN_PROGRESS and COMPLETED. A fresh IN_PROGRESS returns 409 and Retry-After so the double-click does not start a second provider call. A stale IN_PROGRESS (older than the provider timeout budget) resumes the same row.
• COMPLETED replays the stored status and body, including a stored decline.
• Hash mismatch returns 409 and does not call the provider.
• Authorize, capture, void, and refund each use a distinct key. Reusing the authorize key on capture is a hash conflict because the canonical body includes the operation name. That is what we want.
• The provider receives a derived key, principalId + ":" + idempotencyKey, when their documented API has an idempotency mechanism. Derivation keeps one learner's key from colliding with another's in a provider account that is global. The raw client key stays in our row. If the provider documents no idempotency mechanism, the adapter does not blind-retry. The payment stays UNCONFIRMED and reconciliation or a documented status query resolves it.

Keys live for 48 hours in the hot table. A client that retries on day three with the same key is a new ambiguity: we no longer have the stored body. The API returns 409 with a problem type idempotency-key-expired and the client must use a new key only if the order service confirms the order is still unpaid. The order row, not the client's optimism, decides whether a new attempt is safe. This is the coupling that saves you when retention is shorter than human patience.

The ledger is the book, the payment row is the workflow

A payment row answers "what may we do next?" A ledger answers "what do the books say?" Those questions diverge the moment someone ships a bug that sets status = CAPTURED and forgets the money. The repair is painful if the only history is the current balance.

We use double-entry bookkeeping. Each journal has two or more postings. Postings use a signed minor-unit amount in one currency. The sum of postings in a journal is zero. The sum is checked in the application and by a deferred constraint inside the transaction so a journal cannot commit unbalanced. Accounts are rows with a type: asset, liability, income, expense. A debit increases an asset or an expense and decreases a liability or income, which is the ordinary accounting convention; we implement it as signed amounts in a single column (debit_minor positive, credit_minor stored as a negative number, and the journal sums to zero) so the invariant is one arithmetic check.

For a 49.00 USD course (4900 minor units) captured immediately:

Journal S-10492, sale, written when capture commits

The provider fee is not invented at capture time. We do not know the fee with finality until the provider says so. Estimating 2.9% plus 30 cents in the sale journal feels precise and is wrong on regulated debit, on cross-border cards, and on retries that the provider bills differently. The sale journal records the gross the learner owes. The settlement journal records cash and the fee the provider actually took.

Journal SET-8831, settlement, written when a settlement line matches

Assume the provider paid us 4728 and charged a fee of 172:

A refund of the full 4900 before settlement reverses the sale with a new journal. It does not delete S-10492. A refund after settlement debits revenue (or a returns contra-income account if finance prefers the presentation) and credits cash or a refunds-payable account, depending on whether the provider already clawed the cash back. The payment aggregate tracks refunded_minor so the workflow will not refund twice. The ledger tracks the economic event. Both writes commit together.

Manual adjustments, used when reconciliation finds a true provider-side correction, are journals with source = adjustment, an actor, and a reason code. There is no UPDATE ledger_postings.

Accounts are few and stable: receivable per provider account, cash, revenue by product family, processing fees, refunds, a suspense account for settlement lines we cannot match yet. Suspense is how finance sees the break in the trial balance instead of only in a queue nobody opens. A line that sits in suspense past the SLO (two business days in this design) pages the payments on-call and finance together.

Providers are adapters, and failover is a product decision

PaymentProvider from lesson 22 remains the port: authorize, capture, void, refund, and a capability flag for whether the documented API dedupes a key. The high-level addition is a routing policy that chooses a provider code before the use case runs.

Routing inputs we actually have: currency, country of the learner, card brand if the tokenizer returned it as non-sensitive display data, and the merchant's configured preference. Routing output is one provider code. The use case calls that provider. It does not catch a decline and try the next provider. A decline is the issuer's answer. Sending the same charge to a second acquirer is a second authorization hold on the learner's card. That is a product and a scheme-rule problem, not a clever retry.

Failover is allowed only when the first provider returns unavailable and the adapter's capability says the first request was not sent, or the first provider dedupes and we are resuming the same attempt. Opening a second provider for the same order requires a new idempotency key, a void or expiry of any hold that might exist on the first, and an explicit state ROUTE_RETRIED that support can see. This design does not auto-cascade. The interview answer "we fail over to provider B on timeout" is how duplicate holds happen. Say instead: timeout becomes UNCONFIRMED on provider A; a job queries A's documented status endpoint if one exists; only a confirmed absence of a hold permits a new attempt, and that new attempt is a new key.

The HTTP path, host, and secret are @ConfigurationProperties. The course does not invent a vendor resource. A local profile wires ThresholdPaymentProvider, the test double from lesson 22, which approves at or below a configured amount and remembers derived keys in memory. Production does not contain that bean.

Webhooks and settlement are how the outside world corrects us

Providers retry webhooks for days. Our handler:

1. Read the raw body. Signature schemes sign bytes, not a re-serialized JSON object.
2. Verify the signature and the timestamp skew (five minutes is a typical documented tolerance; use the provider's number).
3. Insert (provider_code, provider_event_id) in the same transaction as the state change the event justifies. A unique violation means we already applied it. Commit nothing else and return 2xx so the provider stops retrying.
4. If the handler needs a new provider call, that call sits outside the transaction, with its own idempotency key derived from the event id.

Settlement is a file or a documented payout report, not a webhook we treat as complete. A daily job stores the file checksum so the same file applied twice is a no-op. Each line carries a provider reference, a gross, a fee, a net, a currency, and a value date. Matching is by provider reference to payments.provider_ref. Outcomes:

• Matched and amounts agree: write the settlement journal if it is not already posted (the journal's idempotency key is the provider line id).
• Matched and amounts disagree: suspense journal for the difference, plus a break case. Do not "fix" the payment amount.
• In the file, not in our database: suspense and a break. This is the crash window that never resumed, or a payment another system created on the same provider account.
• In our database as CAPTURED with no line after the provider's settlement lag (three extra business days past their documented lag): a break, not an automatic void. The learner may have been captured and the file may be late.

Consistency choices

Status reads for the learner who just paid come from the primary. Terminal states may be cached briefly and are invalidated when a refund starts. The provider is not a participant in our transaction. Enrollment hears about capture from the outbox and compensates with a refund command, which is lesson 32's saga. The journal and the payment status commit in one PostgreSQL database so a crash cannot publish a capture we did not store. Split the ledger into its own service only when a second product must post, and then post idempotently on journal_key.

Architecture and Design Diagrams

The payment service is the write authority. Checkout and the order service call it. Provider traffic and webhook traffic are separate paths so a replay storm cannot occupy the checkout pool.

mermaidCopy
flowchart LR
  subgraph clients [Clients]
    Browser[Learner browser]
    Order[Order service]
  end

  subgraph edge [Edge]
    CheckoutLB[Checkout L7 load balancer]
    WebhookLB[Webhook L7 load balancer]
  end

  subgraph pay [Payment zone]
    BFF[Checkout BFF]
    PayAPI[Payment API]
    Recon[Reconciliation workers]
    DB[(PostgreSQL primary)]
    Replica[(Sync replica)]
    Cache[(Redis terminal-status cache)]
    Outbox[Outbox relay]
  end

  subgraph outside [Outside trust zone]
    Hosted[Provider-hosted fields]
    Acquirer[Acquirer API]
    Kafka[Event bus]
  end

  Browser --> Hosted
  Browser --> CheckoutLB --> BFF --> PayAPI
  Order --> PayAPI
  PayAPI --> DB
  DB --> Replica
  PayAPI --> Cache
  PayAPI --> Acquirer
  WebhookLB --> PayAPI
  Acquirer --> WebhookLB
  PayAPI --> Outbox --> Kafka
  Recon --> DB
  Recon --> Acquirer

Authorize path, including the crash window lesson 37 described. The provider call is not enlisted in a database transaction.

mermaidCopy
sequenceDiagram
  participant C as Checkout BFF
  participant P as Payment API
  participant DB as PostgreSQL
  participant A as Acquirer

  C->>P: POST /v1/payments Idempotency-Key
  P->>DB: Insert IN_PROGRESS and payment CREATED
  DB-->>P: Commit
  P->>A: Authorize with derived key
  alt Approved
    A-->>P: Approved and provider ref
    P->>DB: AUTHORIZED or CAPTURED, journal, COMPLETED, outbox
  else Declined
    A-->>P: Declined
    P->>DB: DECLINED, COMPLETED, outbox
  else Timeout
    A-->>P: No response
    P->>DB: UNCONFIRMED, COMPLETED with unconfirmed body
  end
  P-->>C: Stored result

Settlement is asynchronous and idempotent. It never mutates a historical journal.

mermaidCopy
flowchart TD
  File[Settlement file] --> Sum[Store file checksum]
  Sum --> Lines[Parse lines]
  Lines --> Match{Provider ref in payments?}
  Match -->|Yes and amounts agree| Post[Post settlement journal if new]
  Match -->|Yes and amounts differ| Break[Suspense plus break case]
  Match -->|No| Break
  Post --> Trial[Trial balance includes cash and fee]
  Break --> Trial

Component responsibilities:

Technical Explanation

Capacity estimate

Assumptions, stated so they can be revised:

• 20 million registered learners. 2 million buyers in a month is a planning peak, not a daily average.
• 1,000,000 payment attempts per day on a launch day (successes, declines, and retries counted separately). Ordinary days are a fifth of that.
• 40% of launch-day attempts arrive in a two-hour window.
• Average of 1.3 HTTP attempts per logical payment because of client retries. The idempotency layer collapses those that share a key.
• Each captured payment writes one payment row, one idempotency row, one sale journal with two postings, one outbox row. Settlement later adds one journal with three postings.
• A status poll fan-out of about five GETs per attempt while the client waits, then rare reads.
• Row size on the order of 1 KB for a payment including indexes as a planning figure, 200 bytes per posting.

Arithmetic:

• Launch-day average rate: 1_000_000 / 86_400 ≈ 12 attempts per second.
• Window rate: 400_000 / 7_200 ≈ 56 attempts per second average during the two hours.
• Burst: three times the window average ≈ 170 attempts per second. Design point: 500 attempts per second sustained so a worse launch and a retry storm still fit. Headroom is deliberate. It is not a claim that we expect 500.
• Provider calls are on the same order as logical attempts, not as HTTP retries, once idempotency works. Budget 200 provider calls per second peak and confirm it against the provider's contracted rate. If the contract is 50 per second, the design must queue or shed before the provider sheds us. Shedding a checkout with 429 and Retry-After, preserving the idempotency key, is acceptable. Opening unlimited threads at the provider is not.
• Database writes at 500 attempts per second if every attempt reached the database: about 500 idempotency inserts plus 500 payment inserts plus, for the captured fraction, journals. If the success rate is 85%, postings are about 0.85 * 500 * 2 ≈ 850 posting inserts per second on the sale path, plus updates. A single PostgreSQL primary with monthly partitions on payments and ledger_postings, synchronous commit, and a second AZ replica is a credible fit at this rate. The ceiling to watch is connection count and lock contention on a hot account row, not raw insert rate.
• Append postings only. A daily balance snapshot plus today's postings is the trial balance. Updating one receivable row on every sale serializes the company.
• Idempotency for two launch days is about 1 GB. A planning year of 50 million captures, four postings, is on the order of 40 GB before indexes. Partition by month. Archive; do not delete.
• Status polls are about five GETs per attempt and hit the primary. Cache only the long tail of terminal reads. A short server-side wait beats an empty poll loop.

The tier that saturates first is the provider contract, then primary database connections if we hold them during the provider call, then disk. Caching does not fix the first. A bulkhead and a short transaction fix the second. That is why lesson 22's "HTTP outside the transaction" rule is a capacity rule as much as a correctness rule.

APIs we own

All commands require Authorization for the caller (learner session via the BFF, or a service credential for the order service) and Idempotency-Key on every POST that can move money or change payment state. Amounts are integers. Currency is ISO 4217. Errors use a problem document with a stable type.

POST /v1/payments

jsonCopy
{
  "order_id": "ord_8f3",
  "amount_minor": 4900,
  "currency": "USD",
  "payment_method_token": "pm_tok_opaque",
  "capture_method": "automatic"
}

The server loads the order, checks principal and amount, and only then proceeds. The token is a bearer secret. It is accepted, hashed for storage, and not written to logs. Response 201 with the payment id, status, and provider reference when one exists. Replay of the same key and hash returns the same status code and body. Hash mismatch returns 409. In-flight duplicate returns 409 with Retry-After.

POST /v1/payments/{id}/capture with body { "amount_minor": 4900 } and a new idempotency key. 409 if the remainder is smaller than the amount or the state is not capturable.

POST /v1/payments/{id}/void with an idempotency key. Legal only from AUTHORIZED with captured_minor = 0.

POST /v1/refunds with { "payment_id", "amount_minor", "reason" } and an idempotency key. The saga derives this key as refund:{refundId}:payment so a retry is not a second refund. Lesson 32's rule, applied here.

GET /v1/payments/{id} reads from the primary. GET /v1/payments/{id}/ledger returns journal ids and postings for support, authorized to finance and to the owning principal in a redacted form (amounts and status, not internal account codes if those are sensitive; in this product they are not secret, but the actor on a manual adjustment is visible only to finance).

POST /v1/provider-webhooks/{providerCode} is unauthenticated by learner identity and authenticated by signature. It is not on the public checkout host. It returns 2xx only after the event is stored or recognized as a duplicate. It returns 401 on a bad signature and stores nothing, so a forged id cannot block the real event later.

POST /v1/reconciliation/settlement-files is an internal upload or a pull trigger. The worker does the matching. Operators do not post journals by hand except through the adjustment API, which requires a step-up role and writes an audit row.

Rate limits, from lesson 35: per principal on create (for example 10 attempts per minute) and per token hash across principals (for example 20 per hour) to blunt stolen-token testing. Limits return 429 with Retry-After and do not consume the idempotency result. A limited request that never started must not be stored as DECLINED.

Schema

sqlCopy
CREATE TABLE payments (
  payment_id          uuid PRIMARY KEY,
  principal_id        uuid NOT NULL,
  order_id            uuid NOT NULL,
  amount_minor        bigint NOT NULL CHECK (amount_minor > 0),
  currency            char(3) NOT NULL,
  status              text NOT NULL,
  capture_method      text NOT NULL,
  authorized_minor    bigint NOT NULL DEFAULT 0,
  captured_minor      bigint NOT NULL DEFAULT 0,
  refunded_minor      bigint NOT NULL DEFAULT 0,
  provider_code       text NOT NULL,
  provider_ref        text,
  token_hash          bytea NOT NULL,
  created_at          timestamptz NOT NULL,
  updated_at          timestamptz NOT NULL,
  CHECK (captured_minor <= authorized_minor),
  CHECK (refunded_minor <= captured_minor),
  CHECK (status <> 'VOIDED' OR captured_minor = 0)
);

CREATE UNIQUE INDEX payments_provider_ref
  ON payments (provider_code, provider_ref)
  WHERE provider_ref IS NOT NULL;

CREATE TABLE idempotency_keys (
  principal_id     uuid NOT NULL,
  idempotency_key  text NOT NULL,
  request_hash     bytea NOT NULL,
  state            text NOT NULL,
  payment_id       uuid,
  response_code    int,
  response_body    jsonb,
  created_at       timestamptz NOT NULL,
  updated_at       timestamptz NOT NULL,
  PRIMARY KEY (principal_id, idempotency_key)
);

CREATE TABLE ledger_accounts (
  account_id    uuid PRIMARY KEY,
  code          text UNIQUE NOT NULL,
  account_type  text NOT NULL,
  currency      char(3) NOT NULL
);

CREATE TABLE ledger_journals (
  journal_id     uuid PRIMARY KEY,
  journal_key    text UNIQUE NOT NULL,
  source         text NOT NULL,
  payment_id     uuid,
  currency       char(3) NOT NULL,
  occurred_at    timestamptz NOT NULL,
  actor_id       uuid,
  reason         text
);

CREATE TABLE ledger_postings (
  posting_id    uuid PRIMARY KEY,
  journal_id    uuid NOT NULL REFERENCES ledger_journals (journal_id),
  account_id    uuid NOT NULL REFERENCES ledger_accounts (account_id),
  amount_minor  bigint NOT NULL,
  CHECK (amount_minor <> 0)
);

A deferred constraint trigger rejects a journal whose postings do not sum to zero. There is no pan or cvv column; a migration test fails the build if either name appears. token_hash is SHA-256 of the payment-method token. Prefer a provider that can resume from provider_ref plus the idempotency key so that bearer token does not need to be stored after the first response. If a provider cannot, keep it encrypted for the payment role only, with the same 48-hour retention as the idempotency row, and never log it.

Webhook dedupe:

sqlCopy
CREATE TABLE provider_events (
  provider_code      text NOT NULL,
  provider_event_id  text NOT NULL,
  received_at        timestamptz NOT NULL,
  payload_hash       bytea NOT NULL,
  PRIMARY KEY (provider_code, provider_event_id)
);

Settlement:

sqlCopy
CREATE TABLE settlement_files (
  file_id      uuid PRIMARY KEY,
  provider_code text NOT NULL,
  checksum     bytea NOT NULL UNIQUE,
  received_at  timestamptz NOT NULL
);

CREATE TABLE settlement_lines (
  line_id          uuid PRIMARY KEY,
  file_id          uuid NOT NULL REFERENCES settlement_files (file_id),
  provider_ref     text NOT NULL,
  gross_minor      bigint NOT NULL,
  fee_minor        bigint NOT NULL,
  net_minor        bigint NOT NULL,
  currency         char(3) NOT NULL,
  match_status     text NOT NULL,
  CHECK (gross_minor = fee_minor + net_minor)
);

CREATE TABLE reconciliation_breaks (
  break_id     uuid PRIMARY KEY,
  line_id      uuid,
  payment_id   uuid,
  break_type   text NOT NULL,
  opened_at    timestamptz NOT NULL,
  resolved_at  timestamptz
);

journal_key values are deterministic: sale:{paymentId}, settlement:{providerCode}:{providerLineId}, refund:{refundId}, adjustment:{adjustmentId}. The unique key makes a re-run of reconciliation insert nothing new.

Scale, cache, and load balancing

The payment API is stateless. An L7 load balancer terminates TLS, uses round-robin or least-requests, and checks /ready, which fails if the primary cannot be reached. Sticky sessions are unnecessary and harmful: a sticky server that dies takes in-flight polls with it for no benefit. Checkout and webhooks use separate listener pools and separate deployments so webhook retries, which arrive in bursts when we have been down, cannot consume the checkout's Tomcat or Netty threads. Reconciliation workers are a third deployment with their own database role that can insert journals and breaks and cannot alter idempotency_keys response bodies.

Connection pools stay small: on the order of dozens per instance, not hundreds, because transactions are short. The provider client has its own bulkhead (a bounded pool of in-flight HTTP calls) and a circuit breaker as in lesson 41. When the breaker is open, create returns 503 with Retry-After and does not insert a payment that pretends to be declined. If the idempotency row was already IN_PROGRESS and we have not sent the provider call, we may leave it for resume; we do not mark DECLINED.

Cache rules:

A cache of COMPLETED responses can sit in front of the database for replay storms, but only as a look-through after the row exists, and a miss must read PostgreSQL. If Redis is flushed, behavior is unchanged.

Read replicas serve finance reports and the reconciliation matcher’s historical scans. They do not serve GET /v1/payments/{id} for the principal who just paid. Replication lag of even a second is a support ticket during checkout.

Partitions: payments and ledger_postings by month on created_at. Queries by primary key include the id; time-based reports prune partitions. If a marketplace of merchants arrives, the shard key becomes merchant_id, and ids are generated so the shard is knowable (a prefix, or a mapping table). Doing that sharding on day one for a single merchant adds a cross-shard journal problem we do not have. Wait.

Security

The PCI boundary is a placement decision. Provider-hosted fields or a tokenizer receive the card. Our document never contains a PAN or a CVV. Logs, traces, and support exports are tested with a denylist: payment_method_token, pan, cvv, cvc, Authorization, provider secret headers. Traces from lesson 43 may include payment_id, order_id, and provider_code. They may include the last four only if we deliberately add that attribute. They must not include the token.

Tokens we do store (encrypted, short-lived) are bearer credentials. Disk encryption alone does not limit which application role can read them. Column encryption with a key in the cloud KMS, decrypted only in the payment process, does. The KMS calls are that cloud's documented encrypt and decrypt operations; the key id is configuration.

Webhook secrets rotate with two active secrets during rotation so a provider still signing with the previous secret is accepted until their console switches. A signature failure is a security event, counted and alerted, and the body is discarded.

Refunds above a threshold require a second person or a step-up role. The threshold is policy (for example anything over 200.00 USD, or any refund on a payment older than 180 days). The system enforces the role; finance sets the number.

Order-to-payment calls use service credentials from lesson 44. The payment service checks that the credential is allowed to act for that order's merchant, which today is TechLearningPro itself. When marketplace payouts arrive, this check is the control that stops one instructor's service account from refunding another's sale.

Failure modes

Enrollment must treat UNCONFIRMED as not paid. Granting the course and reconciling later creates a population of learners we must claw back. Waiting creates a population who paid and wait minutes. The status-query job runs within a minute for UNCONFIRMED rows when the provider documents a query. That is the compromise.

Trade-offs you will be asked to defend

Atomic ledger versus ledger service. One database gives you a balanced journal or no state change. A separate ledger scales independently and becomes a product other flows can post to, at the cost of idempotent consumers and a window where status is CAPTURED and the journal is not yet posted. Choose one database until a second product must post. Revisit when the marketplace payable ledger arrives in the capstone.

Synchronous authorize versus queued authorize. Synchronous fits a checkout button and makes provider latency part of the user path. Queued fits batch renewals: the subscription worker submits with an idempotency key of renewal:{subscriptionId}:{period} and learns the result from the outbox. Both use the same payment API. Do not build a second capture path for renewals.

Automatic versus manual capture. Course access that is immediate uses automatic capture (authorize and capture in one provider conversation if they support it, or capture in the same request flow). A cohort that starts next month can authorize and capture on the start date, and must track authorization expiry, which is a provider rule on the order of seven days for many card schemes and is not something we invent. If we cannot capture before expiry, we re-authorize with a new key and the learner's mandate, or we ask the learner again. Say "the provider's documented expiry" rather than a magic number.

Multi-provider routing versus one provider. One provider is less operational surface and a simpler reconciliation format. A second provider is justified by country coverage or by a contractual outage, and it doubles settlement jobs, secrets, and break codes. Routing is explicit. Decline failover is off.

Storing the token for resume versus requiring provider idempotency. Storing the token widens the secret we must protect. Requiring idempotent authorize shrinks it. Prefer providers whose documented API supports idempotency keys. For a provider that does not, one attempt and a status query is the policy, even if it lowers the automatic recovery rate.

Practical Implementation or Design Walkthrough

This walkthrough follows one purchase of 4900 USD minor units and the refund that follows a failed enrollment. The code uses Spring Framework JdbcClient the way lesson 22 does. The provider HTTP path is a property. The snippet does not name a vendor SDK.

Step 1. Checkout asks the order service for a payable order. The order is PENDING_PAYMENT, amount 4900, currency USD, principal matches the session. The BFF generates nothing about money itself; it forwards an Idempotency-Key the client created and stored in the checkout session so a refresh reuses it. A new key on every page load is a second charge. The BFF's job is to keep the key stable for that attempt.

Step 2. Payment service inserts the idempotency row and a CREATED payment in one short transaction. Unique violation on the key sends the request down the replay path: same hash and COMPLETED returns the body; same hash and fresh IN_PROGRESS returns 409; different hash returns 409.

Step 3. Outside any transaction, the adapter calls the provider with the derived key, the token, the amount, and the currency. ThresholdPaymentProvider is what tests use. HttpPaymentProvider posts to provider.http.authorize-path.

Step 4. On approval with automatic capture, a second transaction sets AUTHORIZED then CAPTURED in the same transaction (both values, captured equal to authorized), inserts journal sale:{paymentId} with postings +4900 receivable and −4900 revenue, marks idempotency COMPLETED, and inserts the outbox row PaymentCaptured. If this transaction fails, resume at step 3 with the same derived key.

Step 5. The relay publishes. Enrollment's inbox inserts event_id and grants the course in one transaction. If enrollment fails terminally (offering withdrawn), the saga posts POST /v1/refunds with key refund:{refundId}:payment.

Step 6. Refund transaction checks refunded_minor + amount <= captured_minor, calls the provider outside the transaction with a derived refund key, then commits refunded_minor, status REFUNDED or PARTIALLY_REFUNDED, journal refund:{refundId}, and the outbox event.

Step 7. Next settlement file matches provider_ref, posts settlement:{lineId} for gross, fee, and net, and closes the receivable. A refund already sent may appear as a negative settlement line. Match it to the refund journal. Do not post the sale settlement as if the refund did not exist.

The journal insert refuses to commit unless postings sum to zero:

javaCopy
public final class JournalWriter {
  private final JdbcClient jdbc;

  public JournalWriter(JdbcClient jdbc) {
    this.jdbc = jdbc;
  }

  public void post(UUID journalId, String journalKey, String currency, List<Posting> postings) {
    long sum = 0;
    for (Posting posting : postings) {
      if (!currency.equals(posting.currency())) {
        throw new IllegalArgumentException("mixed currency in one journal");
      }
      sum += posting.amountMinor();
    }
    if (sum != 0) {
      throw new IllegalArgumentException("journal does not balance");
    }
    jdbc.sql("INSERT INTO ledger_journals (journal_id, journal_key, source, currency, occurred_at) VALUES (?, ?, 'system', ?, now())")
        .params(journalId, journalKey, currency)
        .update();
    for (Posting posting : postings) {
      jdbc.sql("INSERT INTO ledger_postings (posting_id, journal_id, account_id, amount_minor) VALUES (?, ?, ?, ?)")
          .params(UUID.randomUUID(), journalId, posting.accountId(), posting.amountMinor())
          .update();
    }
  }
}

JournalWriter runs inside the caller's transaction. The database trigger is the backstop when a second writer forgets the Java check. Tests cover: unbalanced journal rolls back the payment status change; re-posting the same journal_key is a unique violation treated as success by the reconciliation job; a refund larger than the remainder never calls the provider.

Validation you should be able to describe in a review:

• Contract test that the canonical hash ignores key order and rejects a changed amount.
• Test that a decline is stored and replayed, using ThresholdPaymentProvider above its limit.
• Test that a simulated timeout leaves UNCONFIRMED and does not write a sale journal.
• Test that two concurrent captures with the same key produce one provider call.
• Test that a webhook duplicate returns success and does not double-refund.
• Reconciliation fixture: a file with one match, one amount mismatch, and one unknown reference produces one settlement journal and two breaks.
• ArchUnit or a SQL lint: no column named pan or cvv.

Expected result for the happy path: one payment in CAPTURED, one sale journal that sums to zero, one outbox event, one enrollment, and the next day one settlement journal that clears the receivable. The learner was charged once.

Enterprise Real-World Scenarios

Scenario 1: Launch-day retry storm at a global course company

A large education company opens enrollment for a professional certificate at a published UTC time. Clients on flaky mobile networks time out at 2.5 seconds and retry. The provider's p99 is 2.2 seconds, so a slice of approvals land only on the server after the client has given up.

The design that survives uses a client-held idempotency key, IN_PROGRESS collapsing the duplicate, and a provider that dedupes the derived key. Capacity planning uses the retry multiplier 1.3 on HTTP but 1.0 on provider calls once collapse works. They measure the ratio of provider calls to distinct keys. If that ratio climbs above one, resume is calling the provider twice for keys the provider is not deduping, and the adapter's capability flag is wrong. The fix is to stop blind resume for that provider and move those rows to a status-query job. Webhook ingress is on its own autoscaling group because the provider retries every open event from the three minutes the deployment was restarting. Checkout latency stays flat while webhook CPU spikes. Finance still runs settlement the next morning. The retry storm is invisible in the ledger if and only if journals are keyed by payment id.

What they refuse: a gateway timeout of 30 seconds that holds database connections, and a "retry declined cards once on the backup MID" rule added during the incident. The backup MID double-holds several thousand learners. Reversing those holds is a week of breaks. The post-incident control is a code owner on the routing policy and a test that a decline short-circuits routing.

Scenario 2: A bank-grade audit of a subscription renewal book

A financial-services training vendor sells annual seats to companies. Their auditor asks for the population of renewals in March, the cash that arrived in April, and evidence that refunds were approved by a person. Card numbers are in nobody's extract because the architecture never had them. The extract is a query on ledger_journals joined to payments for source = renewal, plus settlement journals, plus actor_id on refund journals.

Two findings still appear. First, a renewal worker generated a fresh UUID idempotency key on every cron retry, so eleven subscribers have two captures. The ledger shows both, which is why the auditor found them; a balance column would have shown a plausible total. The fix is the key renewal:{subscriptionId}:{period} from lesson 37, and a compensating refund with its own key, not a row delete. Second, one operator updated status to REFUNDED during an incident and did not post a journal. The trial balance still shows the receivable. The fix is to revoke ad-hoc SQL on payments for human roles, require the adjustment API, and add a daily proof query: every CAPTURED payment has a sale:{id} journal, every REFUNDED payment has refund journals summing to refunded_minor. The proof query is the control. The policy document is not the control.

They keep a synchronous replica and test a promotion twice a year. After promotion they run reconciliation before they re-enable capture workers, so a payment that was IN_PROGRESS is resumed deliberately rather than by every replica of the API at once. A distributed lock from lesson 36 around resume, or a conditional update that only one worker can move IN_PROGRESS to a resume lease, prevents a thundering herd of provider calls at failover.

Best Practices

• Copy the amount and currency from the order inside the payment service. Treat the client body as a claim to verify.
• Use integer minor units and one currency per journal. Format for display at the edge.
• Give every money-moving command its own idempotency key, derived deterministically by upstream sagas.
• Commit IN_PROGRESS before the provider call and the business result after it. Keep both transactions short.
• Send the derived idempotency key on the first provider attempt, not only on the retry.
• Store declines as completed results so retries do not hammer the issuer.
• Post journals in the same transaction as the status change they explain, until a ledger service is forced.
• Make journal keys deterministic and postings append-only. Correct with reversing journals.
• Verify webhook signatures on raw bytes, then insert the event id, then apply.
• Separate checkout, webhook, and reconciliation deployments and database roles.
• Cache only terminal learner-facing views. Read the primary for fresh status.
• Put provider calls behind a bulkhead and a timeout. Do not hold a database connection across that wait.
• Reconcile on the provider's cadence. Page on suspense older than the SLO.
• Log payment_id, order_id, status transitions, and the idempotency key. Do not log tokens, PANs, or secrets.
• Test unbalanced journals, duplicate webhooks, timeout mapping, and the "no pan column" lint in CI.
• After any failover, resume UNCONFIRMED and IN_PROGRESS under a lease, then run the settlement matcher before declaring the incident over.
• Write the routing policy down: one provider per attempt, no decline cascade.

Common Mistakes and Anti-Patterns

Treating the provider timeout as a decline. The team maps every exception to DECLINED so the UI has one error state. Learners who were charged see a failure, retry with a new key, and are charged again if the first provider call is not deduped. The status UNCONFIRMED exists so the UI can say "we are confirming" and so enrollment can wait. A decline is a negative authorization response, nothing else.

A single mutable balance. UPDATE accounts SET balance = balance + 4900 is easy to code and impossible to audit. You cannot answer "why" or rebuild after a bad update. Append postings. Derive balances. The extra join is cheaper than the restatement project.

Holding the SQL transaction open across the acquirer HTTP call. Under the launch-day burst the pool hits its max, /ready fails, and the load balancer pulls every instance. The provider may still approve. You have now coupled a latency incident to a correctness incident. Two short transactions, as in the sequence diagram.

Failover on decline. Cascading MIDs or providers after a soft decline multiplies holds. Route before the call. Fail over only when you can show the first attempt did not create a hold.

Webhook dedupe after the side effect, or signature check after the insert. Applying a refund and then inserting the event id double-refunds when the insert fails and the provider retries. Inserting a forged event id before the signature check lets an attacker consume the id and block the real event if you treat any stored id as done. Order is verify, insert, apply, one transaction when the apply step is local.

Logging the payment-method token to debug a 402. The token is a bearer credential. The log pipeline becomes cardholder-adjacent and your PCI scope follows the logs. Log the payment id and the provider's error code. Reproduce with ThresholdPaymentProvider.

Fresh idempotency keys inside cron. A renewal job that does UUID.randomUUID() per attempt converts at-least-once delivery into at-least-once money. Derive keys from the business period.

Reconciling by adjusting the payment amount in place. The settlement file is a statement. The payment is a workflow. Differences go to suspense and to a case. Silent mutation destroys the trail the case needed.

Caching AUTHORIZED. A capture lands and the receipt page still shows authorized for the TTL, or worse, a cache bug serves another learner's payment because the key omitted the principal. Cache terminal views only, keyed by payment id, and authorize the read in the application before the cache lookup so a cache hit is not an authorization bypass.

Troubleshooting and Design Review

Design-review checklist for this system, aligned with Enterprise Architecture Review Checklist:

• Amount is taken from the order, not trusted from the browser.
• Every command has an idempotency story, including the reconciliation poster and the adjustment API.
• UNCONFIRMED cannot capture, cannot refund, and cannot enroll.
• Journal sums are a database invariant, and journal keys are deterministic.
• No PAN, no CVV, token redaction covered by tests.
• Webhook route is separate, signed, and deduped in order.
• Provider routing cannot cascade on decline.
• Capacity numbers name the provider contract as a dependency.
• Failover procedure includes reconciliation before traffic returns.
• Caches cannot authorize and cannot serve non-terminal money state.

Interview Questions and Answers

Q1. A client times out and retries the same payment. What do you store, and what do you send the acquirer?

You store one idempotency row keyed by principal and Idempotency-Key, with a hash of the canonical request, a state, and eventually the response body. The first request commits IN_PROGRESS and a payment id before any provider call. The provider receives a derived key on that first call, not only on the retry, so their documented dedupe has something to find. A retry with the same hash while the row is fresh and IN_PROGRESS gets 409 and Retry-After and does not start a second provider call. A retry after COMPLETED gets the stored body, including a decline. A retry with a different amount gets 409 and no provider call. If the process died after the provider approved and before the second commit, the row is still IN_PROGRESS and stale. Resume calls the provider with the same derived key if, and only if, that provider's documented API returns the original result for a repeated key. Otherwise you mark UNCONFIRMED, you do not enroll, and reconciliation or a documented status query finishes the story. A new UUID on the retry is a second payment. Say that explicitly; it is the mistake renewals make.

Q2. Why is a provider timeout UNCONFIRMED rather than DECLINED or AUTHORIZED?

DECLINED means the issuer or the provider refused the charge. You may show a failure and you must not capture. AUTHORIZED means you have an approval and a provider reference and you may capture within the authorization. A timeout means you do not know which of those happened. Marking it declined invites the client to pay again and produces a double charge whenever the first call succeeded. Marking it authorized enrolls the learner whenever the first call never reached the provider. UNCONFIRMED blocks capture, void, refund, and enrollment. A bounded resume or a status query moves it to a real terminal or authorized state. Settlement is the backstop when the query does not exist. The UI copy is "confirming," not "failed" and not "paid."

Q3. Design the ledger for a capture and a later settlement. Why not update a balance column?

On capture of 4900 minor units, post a journal keyed sale:{paymentId}: debit accounts receivable for the provider 4900, credit revenue 4900. The journal sums to zero or the transaction rolls back, and the payment row becomes CAPTURED in that same transaction. Do not post the processor fee yet. You do not know it. When the settlement line arrives, post settlement:{lineId}: debit cash for the net, debit fee expense for the fee, credit receivable for the gross, and check that gross equals fee plus net on the line. A balance column answers "how much" until the first bug, the first partial refund, or the first auditor, and then it answers nothing because history was overwritten. Append-only postings rebuild the balance by summation or from daily snapshots plus the current day. The snapshot exists to avoid a hot updated row on the single receivable account, which would serialize every capture in the company. Deterministic journal keys make the settlement job safe to rerun.

Q4. Where is the PCI boundary, and what is in your database?

The boundary is in front of our process. The card is collected by provider-hosted fields or a tokenizer. Our API accepts an opaque token. The schema has token_hash, optional encrypted short-lived token material if resume requires it, last four and brand only when the provider returns them as display data, amount, currency, status, and provider reference. It has no primary account number and no card verification value. Logs and traces follow the same rule, enforced by a denylist and by tests. The provider secret lives in a secret store and in configuration for the adapter, not in the payment row and not in git. Webhooks are signed with that provider's documented scheme. This posture is what lets a qualified security assessor consider a short self-assessment questionnaire instead of a full cardholder-data-environment audit. The architecture does not by itself complete compliance; it makes the boundary something an assessor can test. If a developer later posts the card to our BFF "just for validation," the boundary moves and the project is a different system. Say no to that feature in the design review.

Q5. How do you reconcile, and what do you refuse to auto-fix?

Every day, ingest the provider's settlement file, store its checksum, and skip a file you have already applied. For each line, find the payment by provider code and provider reference. If gross, fee, net, and currency agree with the capture you expect, post the settlement journal if that journal key is new. If the reference is unknown or the amounts differ, post the unmatched cash to suspense if cash actually arrived, open a reconciliation_breaks row, and page when suspense ages past two business days. You refuse to change payments.amount_minor to match the file, to delete a journal, or to mark a missing line as void without a provider status that says so. Those edits hide whether the bug was ours, the provider's, or a timing lag. Operators resolve breaks with an adjustment journal that names an actor and a reason. A proof query runs beside the file: every captured payment older than the settlement lag has a sale journal, and every settled reference has a settlement journal. Idempotency prevents double journals. Reconciliation proves the population.

Q6. Can you use a two-phase commit with the acquirer?

No. The acquirer is not a participant in your database's transaction manager. You will not find a supported XA resource for a card authorization in any design you should defend. The practical sequence is a local commit, an HTTP call, and a second local commit, with the provider's idempotency key covering the gap. Outbox publication after the second commit informs enrollment. If you need to undo a capture because enrollment failed, you issue a refund command with its own key. That is a saga, lesson 32, not a two-phase commit. Pretending the provider call sits inside @Transactional only holds pool connections and can roll back your row after their approval. Mention that failure mode; it shows you have operated the system, not only drawn it.

Q7. What do you cache, and what would a cache bug do to money?

Cache terminal payment views for receipt and support reads, with a short TTL and an explicit delete when a refund command starts. Cache last-four display data. Cache rate-limit counters. Do not cache IN_PROGRESS, AUTHORIZED, or UNCONFIRMED as if they were fresh, because the next legal write changes the learner's rights. Do not cache idempotency bodies as the only copy; Redis eviction would re-execute a charge. Do not look up the cache before authorization, or a key collision becomes a data leak. A bug that served another user's cached payment is a confidentiality incident. A bug that served a stale AUTHORIZED after a void can cause a double capture if a worker trusts the cache. Workers read the primary. The cache is a read optimization for data that has stopped changing.

Q8. How do you size the system for a launch, and which tier fails first?

Pick a launch-day attempt count, a peak window, and a retry multiplier, and show the arithmetic. One million attempts, forty percent in two hours, is on the order of fifty attempts per second average in the window and under two hundred at a 3× burst. Designing for five hundred attempts per second gives headroom. Each attempt is a short pair of transactions and one provider call, not a long transaction. PostgreSQL with monthly partitions and append-only postings fits that write rate if you do not maintain a single hot balance row. The provider's contracted requests per second fail first. Above that contract you shed with 429 and keep the client's idempotency key, or you spread launches. The database connection pool fails next if HTTP is inside the transaction. The cache is irrelevant to the write ceiling. State the assumptions. An interviewer who changes the peak to fifty thousand per second is asking for a sharded ledger and a queue in front of the provider, which is a different design you can sketch by naming the shard key (merchant_id) and the new idempotency scope. Do not pretend the single-primary design covers that peak.

Q9. How should multi-provider routing work, and when is failover safe?

Choose the provider from currency, country, and merchant configuration before the call. One attempt talks to one provider. A decline stops. A timeout becomes UNCONFIRMED on that provider. Failover to a second provider is safe only after you have evidence the first attempt created no hold: a documented status query that says not found, or an adapter capability that the request was never sent. The second attempt uses a new idempotency key and is visible as a distinct route attempt on the order. Blind failover on HTTP 500 is how learners get two holds that both settle. Reconciliation then shows two references for one order, which is the detection, not the prevention. Prevention is the routing policy plus the capability flag supportsIdempotentAuthorize from lesson 22. Also separate settlement jobs per provider. A shared parser that assumes one file layout will book the second provider's fees to the wrong account.

Q10. A webhook says captured, and your row says UNCONFIRMED. A settlement line arrives for a payment you cannot find. What do you do in each case?

For the webhook: verify the signature on the raw body first. If the payment row is UNCONFIRMED and the provider reference matches an attempt you sent, the webhook is evidence the provider reached a decision. Apply it in the same transaction as the event-id insert: move to CAPTURED or AUTHORIZED as the event type dictates, and post the sale journal if the event is a capture. If you have no row yet because the webhook won the race with your second commit, store the verified event as pending and let a sweeper apply it when the row exists. Do not return 500 forever if you can store the event; do not return 200 and drop it. For the unknown settlement line: do not synthesize a payment. Post cash to suspense if the file represents money that hit the bank, open a break, and investigate whether a crash lost the row, whether another environment shared the provider account, or whether the reference format changed. Synthesizing a CAPTURED payment would enroll nobody and would train the team to trust the file over the workflow. The break queue is the product.

Q11. How do refunds interact with partial capture, chargebacks, and the enrollment saga?

The aggregate allows a refund only up to captured_minor - refunded_minor, and only from a captured or partially refunded state. Each refund command has its own idempotency key. Partial capture reduces what may later be refunded because you cannot refund what you never captured; you void the remainder instead. The enrollment saga's compensation calls this refund API. It does not open a second code path that updates enrollment and payment in one private transaction. A chargeback is the scheme clawing back a capture. Record it as its own journal and reduce the refundable remainder so support cannot refund money the bank already returned to the learner. If both a refund and a chargeback land, the second one hits a remainder of zero and becomes a break, not a negative refund. That conservative refusal is intentional. The policy for who eats the loss is a finance decision recorded as an adjustment, with an actor.

Q12. What will you say in the Module 6 design review when someone asks for 99.999% and a 200 ms end-to-end authorize?

You will separate our overhead from the provider's latency and from enrollment. Our API can budget 150 ms p99 excluding the provider. The provider call is often the majority of a multi-second tail, and no load balancer changes the issuer. 99.999% of successful authorizations is a provider-dependency SLO you do not control; you publish our availability excluding provider outages and a learner-impact SLO that includes them, so product sees the truth. You will show RPO 0 for committed journals via synchronous replication, and RTO measured by a promotion drill, not by a slide. You will show the proof query and the suspense age. You will point at Architecture Performance and Capacity Review for the arithmetic and at Architecture Security and Threat Modeling for the token and webhook threats. If they still want 200 ms end to end, the design changes to asynchronous capture confirmation, and the UI stops pretending the button completed a network authorization. That is a product change, and you will say so before you redraw boxes.

Hands-On Exercises

Exercise 1: Specify the payment API and the journal set

Problem. TechLearningPro will sell a 49.00 USD course and a 120.00 USD team seat. Team seats capture only when the buyer assigns the first learner, up to seven days later. Design the commands, states, and journals. Provider HTTP paths stay configuration. Do not invent a vendor resource.

Tasks.

1. List functional and non-functional requirements, including authorization expiry as "whatever the configured provider documents," not a made-up scheme rule.
2. Write the request and response for create, capture, and refund, including idempotency behavior for replay, in-flight duplicate, and hash conflict.
3. Draw the state machine. Show why a team seat stays AUTHORIZED and a course becomes CAPTURED.
4. Write the sale, settlement, and refund journals for the 4900 minor-unit course with a fee you only learn at settlement (use 172 as the fixture fee).
5. Specify one crash between provider approval and local commit, and the exact resume rule.
6. Specify webhook verification order and settlement break types.
7. Estimate writes per second at 200 attempts per second with 90% approval.
8. Name two caches you refuse to add.

Deliverables. A short design note, one sequence diagram, the journal tables, and a failure table.

Acceptance. A reviewer can point to the idempotency key on each command, show that journals balance, show that UNCONFIRMED does not enroll, and show that the fee is absent from the sale journal. The team-seat path does not capture inside the create call.

Exercise 2: Review a flawed design

You are handed this proposal. "Store the card number encrypted in payments.pan_encrypted. On any exception, mark the payment failed and retry with provider B. Update accounts.balance in the same statement as the status. Put the provider call inside @Transactional so it all rolls back. Skip reconciliation because idempotency keys guarantee exactly-once."

Write a review that rejects each sentence with the consequence and the replacement control. Include a proof query you would add. This is the checklist exercise for Enterprise Architecture Review Checklist.

Rubric.

Lesson Summary

TechLearningPro Payments keeps the lesson 22 state machine and the lesson 37 idempotency record, and it adds the system around them: an order-checked amount, a provider port whose failover rules refuse decline cascades, a double-entry ledger posted in the same transaction as the workflow change, signed and deduped webhooks, and a settlement reconciliation that opens breaks instead of rewriting history. Capacity at a few hundred attempts per second fits a single PostgreSQL primary if transactions stay short and balances are not a hot row. The provider contract saturates before the database does. The PCI boundary is physical: card data never arrives, so it cannot leak into logs. UNCONFIRMED is the honest timeout state, and enrollment waits. Journals are append-only and deterministic so a retrying worker cannot invent money.

The design is right for a merchant learning product. It is the wrong shape if you need to issue cards, if you must shard by merchant already, or if a second product must post to the ledger and you have not yet built an idempotent journal consumer. Marketplace payouts belong to the capstone, using the same posting rules and a new payable account. Module 6's interview framework, SLO lesson, threat model, disaster-recovery drill, and architecture checklist are where you defend the numbers and the boundary under questions this lesson already rehearsed.

Further Reading

• LLD Case Study: Payment Gateway and Idempotency and Exactly-Once Processing, the two lessons this case study sits on.
• Distributed Transactions and Saga Pattern for refund compensation across enrollment.
• Non-Functional Requirements and SLO Design, Architecture Security and Threat Modeling, Disaster Recovery and Business Continuity, Architecture Performance and Capacity Review, System Design Interview Framework, and Enterprise System Design Capstone Project.
• IETF draft, The Idempotency-Key HTTP Header Field: https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/.
• Stripe's public idempotency documentation, as an example of a provider-documented key, not as a contract this course invents: https://docs.stripe.com/api/idempotent_requests.
• PCI Security Standards Council document library, for the boundary and the assessment your QSA actually uses: https://www.pcisecuritystandards.org/.
• Martin Fowler's accounting narrative on events and positions, the conceptual ancestor of append-only journals: https://martinfowler.com/eaaDev/AccountingNarrative.html.
• ISO 20022, the family of financial messages settlement files often resemble: https://www.iso20022.org/.
• PostgreSQL CREATE TABLE and constraint triggers, for the journal balance backstop: https://www.postgresql.org/docs/current/sql-createtable.html.