Idempotency

Introduction

Idempotency ensures repeated identical requests produce the same outcome as a single execution — the foundation of safe retries in distributed systems. Stripe pioneered the Idempotency-Key HTTP header for payment APIs; the pattern now spans sagas, webhooks, Kafka consumers, and mobile clients with flaky networks.

Real production story

Stripe's Connect onboarding API saw mobile clients retry POST /accounts on slow networks. Without idempotency, duplicate requests created twin connected accounts — merchants saw double verification emails and split payout history. Support volume spiked; fraud rules flagged "duplicate identity" false positives.

Engineering shipped mandatory Idempotency-Key: server stores request fingerprint + response for 24 hours; replays return cached 200 with original body. Duplicate account creation dropped to zero measurable rate. The pattern became public API documentation and industry standard for all mutating payment endpoints.

Business problem

Business pressure: Stripe API clients retry aggressively — networks, load balancers, and SDKs all re-send. Without idempotency, money movement APIs cannot be safely retried, blocking reliable integrations.

• Financial correctness: Double charges destroy merchant trust and trigger card network fines.
• Developer experience: Integrators expect "retry until 200" semantics on POST — idempotency makes that safe.
• Internal systems: Saga steps and Kafka consumers also retry — same semantics required server-side.

Architecture overview

Idempotency key = client-supplied unique token (UUID) scoped to operation. Server records (key → request hash, response, status) atomically before side effects. Natural idempotency = operations like PUT by ID that are safe without extra store.

• Definition: f(f(x)) = f(x) at business level — duplicate invocations do not duplicate effects.
• When to adopt: All mutating APIs, payment flows, saga steps, message consumers.
• When natural: SET balance=100 is not idempotent; INCREMENT is not — need keys or dedup table.
• Operability: Metrics on replay hit rate, conflict rate, store latency.

Architecture motivation

Why architects care: At-least-once delivery is the default in distributed systems; idempotency converts it to effectively-once business behavior.

• Force: HTTP timeouts leave client uncertain if server processed request — retries are mandatory.
• Constraint: Idempotency store must be durable and faster than primary business logic.
• Outcome: Standard key format, TTL policy, and conflict detection (same key, different body → 409).

Internal architecture

Stripe Idempotency-Key middleware:

Client: POST /v1/charges
        Headers: Idempotency-Key: uuid-v4
                 Authorization: Bearer sk_...

        ↓
   API Gateway → IdempotencyMiddleware
        │
        ├─ GET idempotency_store[key]
        │     hit + same body hash → return cached response
        │     hit + diff body hash → 409 Conflict
        │     miss → BEGIN TX
        │              INSERT idempotency (key, status=IN_PROGRESS)
        │              run ChargeHandler
        │              UPDATE status=COMPLETED, response_body=...
        │            COMMIT
        ↓
   Response identical on every retry

Data flow

Idempotency check precedes irreversible side effects.

• Write path: Reserve key IN_PROGRESS → execute → store COMPLETED response.
• Read path: GET by key returns cached response without re-executing.
• Conflict path: Same key, different payload → 409 — client bug, not silent corruption.

System design diagram

Two diagrams show the Idempotency topology and the primary request/event path used in production at scale.

Production code example

Idempotency middleware — Stripe-style request deduplication:

async function withIdempotency(
  key: string,
  requestHash: string,
  handler: () => Promise<ApiResponse>,
): Promise<ApiResponse> {
  const existing = await idempotencyStore.get(key);
  if (existing?.status === "COMPLETED") {
    if (existing.requestHash !== requestHash) {
      throw new ConflictError("Idempotency key reused with different parameters");
    }
    return existing.response;
  }
  if (existing?.status === "IN_PROGRESS") {
    throw new ConflictError("Request in progress", { retryAfterMs: 500 });
  }

  await idempotencyStore.insert({ key, requestHash, status: "IN_PROGRESS" });
  try {
    const response = await handler();
    await idempotencyStore.complete(key, response);
    return response;
  } catch (err) {
    await idempotencyStore.fail(key, err);
    throw err;
  }
}

Enterprise case study

Stripe Idempotency-Key — industry-standard safe retries for payment APIs.

• Before: Duplicate Connect accounts from mobile retries; support tickets + fraud noise.
• Decision: Mandatory header on mutating endpoints; 24h durable store; 409 on body mismatch.
• After: Zero duplicate account creation; integrator docs cite pattern globally.

Trade-offs

• Storage vs safety: 24h+ retention of responses increases Redis/DB cost — mandatory for payments.
• IN_PROGRESS handling: Concurrent duplicate requests while first still running — lock or return 409 with Retry-After.
• Key scope: Too-coarse keys block legitimate distinct ops; too-fine keys miss dedup.
• Non-HTTP: Kafka consumers need messageId or business-key dedup table — header pattern alone insufficient.

Security considerations

Idempotency keys are not auth — they prevent duplication, not impersonation.

• Scope keys per account: Key namespace includes merchant ID — prevent cross-tenant replay.
• Do not leak responses: Cached error bodies may expose internal details — sanitize.
• Rate limit key creation: Prevent attacker filling idempotency store.

Scalability analysis

Stripe-scale idempotency is a hot path — sub-millisecond lookups required.

• Horizontal scale: Shard idempotency store by hash(key); sticky routing optional.
• Hot keys: Buggy client reusing one key — rate limit + alert on replay storm.
• Cost: TTL expired keys to cold storage; compress response bodies.

Failure scenarios

Crash between side effect and store update is the classic idempotency bug.

• Charge succeeded, store write failed: Retry finds no key → double charge — use outbox or two-phase store update before external call where possible.
• Stuck IN_PROGRESS: TTL expires; second request may duplicate — heartbeat or lease on in-progress rows.
• Store partition: Fail closed (503) rather than proceed without dedup.

Staff engineer insights

• Idempotency before external side effect — not after. If money left the building, store must already have the key.
• 409 on same-key-different-body catches client bugs early — treat as gift, not annoyance.
• Consumers: dedup table with (messageId, handler) unique index beats "hope Kafka is once".

Interview questions

Architecture review questions

• All mutating endpoints accept idempotency key or have natural idempotency documented.
• Key reservation atomic before external side effects (payment, email, inventory).
• 409 returned on key reuse with different request body.
• IN_PROGRESS timeout and cleanup policy defined.
• Idempotency store scoped per tenant/account — no cross-customer key collision.
• Metrics: replay rate, conflict rate, store p99 latency.

Summary

Idempotency converts unreliable networks and message brokers into safe retry semantics. Stripe's production Idempotency-Key design — durable store, conflict detection, and before-side-effect reservation — is the template for enterprise mutating APIs and event consumers.