Distributed Transactions

Introduction

Distributed transactions coordinate commits across multiple services or databases. At Amazon scale, classical 2PC/XA is avoided in favor of sagas, idempotent workers, and purpose-built stores (DynamoDB transact writes within a partition). This lesson covers when distributed ACID is feasible, why most microservice architects reject global transactions, and what Amazon uses instead.

Real production story

An Amazon retail team attempted XA transactions between inventory DynamoDB and payment RDS via a Java EE transaction manager. Black Friday lock contention froze checkout for 18 minutes in one region — 2PC prepare phase held partition locks while payment gateway latency spiked. Rollback storms amplified DynamoDB throttling.

Principal engineers mandated: no cross-service 2PC. Inventory reservation moved to DynamoDB transactWriteItems within one table; cross-service coordination uses Step Functions sagas with idempotent compensations. Checkout recovered; p99 stayed flat under 3× traffic. The ADR became template guidance for all marketplace teams.

Business problem

Business pressure: Amazon checkout must reserve inventory, charge payment, and create shipment plans atomically from the customer's perspective — without halting the site when one datastore hiccups.

• Availability: Global 2PC coordinators become single points of failure during peak.
• Latency: Prepare/commit rounds add RTT across regions — unacceptable for sub-second checkout.
• Organizational: Inventory and payments are different services with independent deploy cycles — shared TM coupling violates team autonomy.

Architecture overview

Local transactions remain ACID within one database or DynamoDB transact boundary. Global coordination uses sagas, outbox, and deterministic idempotency — not blocking locks across teams.

• Definition: Cross-service atomicity via choreography of local commits + compensations.
• When 2PC appears: Rare — single vendor stack (e.g., Spanner external transactions) with ops maturity.
• When to avoid: Cross-team microservices, multi-region, heterogeneous stores — default no.
• Operability: Reconciliation jobs detect drift; metrics on in-doubt transaction count (should be zero).

Architecture motivation

Why architects care: True distributed ACID across arbitrary services conflicts with CAP under partition — Amazon chooses partition-tolerant patterns with explicit inconsistency windows.

• Force: Multi-item checkout spans three bounded contexts and two database engines.
• Constraint: Cannot regress availability SLO for strong global consistency.
• Outcome: Local ACID + saga/orchestration + idempotency replaces global 2PC.

Internal architecture

Amazon checkout — no global 2PC:

CheckoutAPI
    ↓
Step Functions (CheckoutSaga)
    ├─ Task: ReserveInventory
    │     DynamoDB TransactWrite (item row + reservation row)
    ├─ Task: AuthorizePayment (idempotent)
    │     PaymentService → external gateway
    ├─ Task: CreateShipmentPlan
    └─ on failure → CompensateInventory + VoidAuth

❌ Avoided: XA TM spanning DynamoDB + RDS
✅ Local: transactWriteItems within inventory table
✅ Cross: saga with timeouts + DLQ for stuck runs

Data flow

Each leg commits locally; saga state tracks global progress.

• Write path: Reserve inventory in single DynamoDB transaction; emit InventoryReserved via outbox.
• Payment path: Idempotent authorize with checkoutId key; void on compensation.
• Reconcile path: Nightly job compares reservations vs payments vs shipments.

System design diagram

Two diagrams show the Distributed Transactions topology and the primary request/event path used in production at scale.

Production code example

DynamoDB local transaction + saga step:

async function reserveInventory(checkoutId: string, items: LineItem[]): Promise<void> {
  const transactItems = items.flatMap((item) => [
    {
      Update: {
        TableName: "Inventory",
        Key: { sku: item.sku },
        UpdateExpression: "SET available = available - :qty",
        ConditionExpression: "available >= :qty",
        ExpressionAttributeValues: { ":qty": item.quantity },
      },
    },
    {
      Put: {
        TableName: "Reservations",
        Item: { checkoutId, sku: item.sku, qty: item.quantity, ttl: ttl24h() },
        ConditionExpression: "attribute_not_exists(checkoutId)",
      },
    },
  ]);

  await dynamodb.transactWrite({ TransactItems: transactItems }).promise();
  await outbox.emit({ type: "InventoryReserved", checkoutId, items });
}

// Step Functions task — idempotent via checkoutId

Enterprise case study

Amazon marketplace checkout — retired cross-store XA in favor of local transact + Step Functions.

• Before: 18-minute regional checkout freeze; 2PC lock pile-up on Black Friday.
• Decision: Ban cross-service 2PC; standardize checkout saga template.
• After: p99 checkout stable at 3× load; zero XA coordinators in path.

Trade-offs

• Strong global ACID vs availability: 2PC blocks on failure — Amazon prioritizes checkout uptime.
• Saga complexity vs coupling: Orchestration code replaces TM but needs testing and monitoring.
• Transient inconsistency: Inventory reserved but payment pending visible internally — UX shows "processing".
• DynamoDB transact limits: 25 items, same account — shapes aggregate design.

Security considerations

Payment and inventory APIs are fraud targets — saga orchestrator credentials are crown jewels.

• AuthZ: Each saga step validates checkout token scoped to customer session.
• Idempotency: Prevents replay attacks doubling charges or reservations.
• Audit: Immutable saga execution history for chargeback disputes.

Scalability analysis

Black Friday stresses reservation hot keys — partition design beats global locks.

• Horizontal scale: Saga workers scale independently; DynamoDB on-demand for reservation table.
• Hot SKUs: Oversell protection via conditional writes, not global locks.
• Cost: Step Functions state transitions priced per checkout — optimize step count.

Failure scenarios

Partial checkout states are normal — architecture defines detection and cleanup.

• Payment timeout after reserve: Saga compensates inventory release; alert if compensation fails.
• Duplicate checkout submit: Idempotency-Key on CheckoutAPI returns same result.
• Step Functions outage: Stuck sagas resume from last checkpoint; never double-charge.

Staff engineer insights

• "We need distributed transactions" usually means "we need a saga and haven't named it yet."
• DynamoDB transactWrite is not a license for cross-service ACID — stay within one aggregate table.
• If reconciliation job is missing, you do not have distributed transactions — you have hope.

Interview questions

Architecture review questions

• No cross-service 2PC/XA — ADR documents alternative (saga/local transact).
• Each saga step idempotent with deterministic keys derived from business ID.
• Compensation paths tested under chaos (kill worker mid-step).
• Nightly reconciliation compares cross-service invariants.
• DynamoDB transact items stay within 25-item and same-account limits.
• Checkout UX covers in-progress saga state without duplicate submit.

Summary

Distributed transactions in enterprise microservices mean orchestrating local ACID boundaries — not global two-phase commit. Amazon's checkout saga pattern replaces XA with Step Functions, idempotency, and DynamoDB transact writes for reliable marketplace scale.