Reliability

Introduction

Reliability is the probability that a system performs correctly over time — correctness under failure, not just uptime. Stripe payment infrastructure must never lose or duplicate money movement; reliability architecture prioritizes idempotency, audit trails, and deterministic recovery over raw availability theater.

Real production story

A Stripe Connect payout batch partially completed when a worker crashed mid-chunk. Retries without idempotency keys duplicated two payouts to the same connected account — caught within minutes by reconciliation, but trust impact was real. The post-incident shift treated reliability as correctness architecture: every mutating API requires idempotency keys, outbox for side effects, immutable ledger events, and hourly parity checks between ledger and bank files. Duplicate payout class incidents dropped to zero over eighteen months.

Business problem

Stripe moves billions daily. Reliability failures are not "503 for five minutes" — they are incorrect balances, duplicate charges, and regulatory exposure. Business requires provable correctness, not best-effort delivery.

• Financial correctness: Every cent must be accounted for — lost updates are unacceptable.
• Idempotent recovery: Retries are inevitable; architecture must make them safe by design.
• Audit and compliance: Regulators and customers require explainable, replayable transaction history.

Architecture overview

Reliability encompasses correctness, recoverability, and maintainability under failure. Staff architects wire invariants into the data model, use outbox/saga for distributed writes, and treat reconciliation as a first-class subsystem.

• Idempotency: Same request applied twice yields same outcome — keys on all mutating APIs.
• Immutability: Append-only ledger; corrections via compensating entries, not silent updates.
• Deterministic replay: Event log rebuilds state — disaster recovery is architecture, not backup tape hope.
• Verification: Continuous reconciliation between internal ledger and external sources of truth.

Architecture motivation

Reliability engineering at Stripe merges SRE availability with financial systems engineering: ledger invariants, exactly-once semantics where required, and at-least-once with idempotency everywhere else.

• Force: Network partitions and worker crashes are normal — correctness cannot depend on perfect execution.
• Constraint: Cannot serialize all payments through one global lock — scale requires partition-tolerant design.
• Outcome: Reconciliation detects drift before customers; replay reconstructs state from events.

Internal architecture

Stripe reliability core — ledger-centric with async side effects:

• State machines make partial failure explicit — no ambiguous "processing" forever.
• Outbox pattern ties DB commit to side effect delivery — no dual-write races.

API Gateway (idempotency-key header required)
   ↓
Payment orchestrator (state machine)
   ↓
Ledger (append-only, double-entry invariants)
   ↓
Outbox table → relay → webhooks / bank files
   ↓
Reconciliation jobs (hourly parity)
   ↓
Immutable audit log (compliance retention)

Data flow

Write path reliability: Validate idempotency → persist ledger entry in same transaction as outbox row → async relay with retry → reconciliation verifies downstream.

• Charge: Idempotency lookup → reserve → capture → ledger append → outbox webhook.
• Retry: Same idempotency key returns original response — never double-charge.
• Recovery: Replay uncommitted outbox rows; saga compensates failed bank submissions.

System design diagram

Two diagrams show the Reliability topology and the primary request/event path used in production at scale.

Production code example

Idempotent charge handler — TypeScript production pattern:

• Idempotency record and ledger append in one transaction — atomic reliability boundary.
• Outbox insert in same transaction guarantees webhook eventual delivery.

export async function createCharge(
  db: Database,
  req: ChargeRequest,
  idempotencyKey: string
): Promise<ChargeResult> {
  return db.transaction(async (tx) => {
    const existing = await tx.idempotency.find({
      merchantId: req.merchantId,
      key: idempotencyKey,
    });
    if (existing) return existing.response as ChargeResult;

    const charge = await tx.ledger.appendCharge({
      merchantId: req.merchantId,
      amount: req.amount,
      currency: req.currency,
      idempotencyKey,
    });

    await tx.outbox.insert({
      aggregateId: charge.id,
      eventType: "charge.created",
      payload: charge.toJSON(),
    });

    const result = { chargeId: charge.id, status: charge.status };
    await tx.idempotency.save({
      merchantId: req.merchantId,
      key: idempotencyKey,
      response: result,
    });
    return result;
  });
}

Enterprise case study

Stripe Connect payout reliability overhaul: Batch processing without idempotent chunks caused duplicate payouts under crash-recovery.

• Before: Manual reconciliation caught most drift; two duplicate payout Sev-1s per year.
• Decision: Chunk-level idempotency, outbox for bank files, automated hourly parity dashboard.
• After: Zero duplicate payouts in 18 months; MTTR for ledger drift under 30 minutes.

Trade-offs

• Strong consistency vs throughput: Ledger serialization per account limits QPS — shard accounts, not invariants.
• Sync vs async confirmation: Merchants want instant webhooks; async improves reliability — document delivery guarantees.
• Storage cost vs audit: Immutable logs grow forever — tiered storage with legal retention policies.
• Complexity vs manual ops: Reconciliation automation is expensive upfront — cheaper than incident response.

Security considerations

Reliability intersects security: Forged idempotency keys and replay attacks target payment APIs — authn/authz on every mutating call.

• Key scope: Idempotency keys bound to merchant + endpoint — prevent cross-merchant replay.
• Ledger integrity: Cryptographic chaining or WORM storage for tamper evidence.
• Secrets rotation: Bank connector creds rotated without dropping in-flight sagas — versioned credentials.

Scalability analysis

Reliable systems must scale: Idempotency stores, ledger partitions, and reconciliation throughput grow with payment volume.

• Idempotency TTL: Keys retained 24–72h minimum; storage partitioned by merchant.
• Ledger sharding: Shard by account ID; cross-account transfers use two-phase ledger entries.
• Reconciliation parallelism: Map-reduce parity checks; alert on drift thresholds, not single-row diffs only.

Failure scenarios

Reliability failures Stripe designs against: duplicate processing, lost messages, clock skew, and partial saga completion.

• Duplicate payout: Missing idempotency on batch worker — keys on every chunk boundary.
• Lost webhook: Outbox without relay monitoring — alert on outbox age p99.
• Split saga: Bank accepted, ledger not updated — reconciliation triggers compensating entry.

Staff engineer insights

• Availability without correctness is worthless for payments — 200 OK that loses money is worse than 503.
• If retries are not idempotent by design, your system will duplicate production traffic during every incident.
• Reconciliation is not a batch job shame — it is the proof your architecture actually works.

Interview questions

Architecture review questions

• Are all mutating APIs idempotent with scoped keys?
• Is ledger append-only with compensating entries for corrections?
• Does outbox pattern tie side effects to DB commits?
• Are saga state machines documented with compensation paths?
• Is continuous reconciliation running with alert thresholds?
• Can state be rebuilt from event log within defined RPO?

Summary

Reliability at Stripe means architecting for correct money movement under inevitable failures: idempotent APIs, append-only ledgers, outbox delivery, saga compensation, and continuous reconciliation — because retries will happen and duplicates are unacceptable.