Saga Pattern

Introduction

Saga pattern coordinates long-running business transactions across multiple services without a distributed two-phase commit. Each local transaction publishes an event or command; compensating transactions undo prior steps when a downstream leg fails. Netflix Conductor and similar orchestrators formalize this at scale — choreography via events when coupling is loose, orchestration when audit trails and timeouts matter.

Real production story

During a global content-licensing rollout, Netflix's subscription billing team split monolith checkout into Order, Billing, Entitlement, and Notification services. A naive "call each service synchronously" path left thousands of users charged but without streaming access when Entitlement timed out after Billing succeeded. Finance could not auto-refund because Order state read "pending" while Stripe webhooks showed "captured."

The post-incident design adopted an orchestrated saga: Order Service creates a saga instance; each step is idempotent with a defined compensating action (refund, revoke license, cancel email). Conductor dashboards expose per-saga state; SREs replay failed compensations instead of hand-editing five databases. Charge-without-entitlement incidents dropped to near zero within one quarter.

Business problem

Business pressure: Netflix must activate subscriptions and refunds across dozens of regional payment rails within seconds of user action. A single ACID transaction spanning Billing (US), Entitlement (EU cache), and CRM (Salesforce) is impossible — yet finance and legal require explainable, reversible multi-step flows.

• Revenue integrity: Partial success (money captured, no access) generates support cost and regulatory exposure in EU consumer-protection regimes.
• Release velocity: Teams cannot ship new pricing tiers if every feature requires a new distributed lock protocol.
• Auditability: Finance needs a durable timeline of saga steps and compensations, not grep across five log systems.

Architecture overview

Saga = sequence of local transactions where each step commits independently. Choreography: services react to events (loose coupling, harder debugging). Orchestration: central coordinator issues commands (clear state machine, single place for timeouts).

• Definition: A long-lived business process decomposed into compensatable local transactions.
• When to adopt: Multi-service workflows with no single natural aggregate root (checkout, onboarding, migration).
• When to defer: Single bounded context — use one database transaction instead.
• Operability: Saga ID in every log line; dashboard showing running/completed/compensating counts.

Architecture motivation

Why architects care: Sagas trade global ACID for eventual business consistency with explicit failure semantics. The alternative — synchronous RPC chains with manual cleanup — fails silently under timeouts and retry storms.

• Force: Microservices own their data; no shared transaction coordinator across AWS regions.
• Constraint: Compensations must be legally valid (refunds, not silent deletes) and idempotent.
• Outcome: Each saga step has an owner, timeout, retry policy, and compensating handler documented in ADR-1847.

Internal architecture

Netflix-style orchestrated saga — coordinator owns state machine; workers are stateless:

Client POST /subscribe
        ↓
   Order API (creates sagaId, returns 202)
        ↓
┌──────────────────────────────────────┐
│  Saga Orchestrator (Conductor-like)  │
│  state: RESERVE → CHARGE → ENTITLE   │
└──────────┬───────────┬───────────────┘
           ↓           ↓
    Billing Worker   Entitlement Worker
    (local TX)       (local TX + cache)
           ↓           ↓
    on failure ←── CompensatingCommand
           ↓
    Saga audit log (DynamoDB / RDS)
           ↓
    Observability: sagaId trace, step latency

Data flow

Happy path: orchestrator loads saga definition, invokes step N with idempotency key, waits for callback or polls status topic, advances or triggers compensation.

• Write path: Step handler updates local DB + publishes StepCompleted(sagaId, step, payloadHash).
• Read path: Client polls GET /sagas/{id} or receives webhook on terminal state.
• Async path: Failed steps enqueue CompensateStep(sagaId, fromStep) with exponential backoff.

System design diagram

Two diagrams show the Saga Pattern topology and the primary request/event path used in production at scale.

Production code example

Orchestrated saga worker — idempotent step execution with compensation registry:

interface SagaStepHandler {
  execute(ctx: SagaContext): Promise<StepResult>;
  compensate(ctx: SagaContext): Promise<void>;
}

class ChargeBillingStep implements SagaStepHandler {
  async execute(ctx: SagaContext): Promise<StepResult> {
    const key = `saga:${ctx.sagaId}:charge`;
    const existing = await this.store.get(key);
    if (existing) return existing;

    const charge = await this.billing.charge({
      customerId: ctx.customerId,
      amount: ctx.amount,
      idempotencyKey: key,
    });
    const result = { status: "CHARGED", chargeId: charge.id };
    await this.store.put(key, result);
    await this.audit.log({ sagaId: ctx.sagaId, step: "CHARGE", result });
    return result;
  }

  async compensate(ctx: SagaContext): Promise<void> {
    const key = `saga:${ctx.sagaId}:charge`;
    const prior = await this.store.get(key);
    if (!prior?.chargeId) return;
    await this.billing.refund({
      chargeId: prior.chargeId,
      idempotencyKey: `${key}:refund`,
    });
  }
}

Enterprise case study

Netflix subscription activation saga — orchestrated flow replacing synchronous RPC chain after charge-without-entitlement incident.

• Before: 0.3% of signups required manual reconciliation; mean time to fix 4.2 hours.
• Decision: Conductor-style orchestration with explicit compensateRefund and compensateRevoke steps.
• After: Automated compensation success rate 99.97%; finance audit exports saga timeline per charge ID.

Trade-offs

• Consistency vs simplicity: Sagas accept temporary inconsistency between steps — UX must communicate "processing" states.
• Choreography vs orchestration: Event-only sagas scale team autonomy but make global timeouts and ordering harder to enforce.
• Compensation cost: Not every business action is reversible (email sent, DRM key issued) — design "pivot" or manual intervention steps.
• Storage overhead: Durable saga state and audit logs grow with every checkout — retention and GDPR erasure policies required.

Security considerations

Security is architectural: Saga orchestrator is a high-privilege component — compromise allows arbitrary refunds or entitlement grants.

• Identity: mTLS between orchestrator and workers; OAuth client credentials scoped per step type.
• Data: Saga payloads minimize PII; encrypt saga audit at rest; TTL for completed instances.
• Authorization: Only Order service may start subscription sagas — validate caller + business context.

Scalability analysis

Scale dimensions: Netflix-scale signup spikes create millions of concurrent sagas; orchestrator and saga store must shard by tenant or sagaId.

• Horizontal scale: Stateless workers autoscale; orchestrator partitions saga instances by hash(sagaId).
• Hot spots: Global promo events concentrate sagas on Billing — bulkhead worker pools per step type.
• Cost: Polling clients vs push webhooks — prefer event-driven completion to reduce read amplification.

Failure scenarios

What breaks: duplicate step delivery, compensating action failure, and orchestrator partition during peak.

• Duplicate step execution: Idempotency keys on every handler; saga log records step completion exactly once.
• Compensation failure: Dead-letter queue + human workflow — never leave saga in "compensating" forever without alert.
• Poison saga definition: Bad deploy registers invalid state machine — feature-flag saga versions and canary new definitions.

Staff engineer insights

• If you cannot name the compensating action for step N, step N should not exist — sagas are not "retry until lucky."
• Orchestrator timeouts must be shorter than client timeouts — otherwise users abandon while saga still runs.
• Netflix-scale sagas need a "stuck saga" SLO: alert when any instance exceeds 2× p99 step duration.

Interview questions

Architecture review questions

• Every saga step has a documented compensating action or explicit "non-reversible" flag with manual runbook.
• Idempotency keys scoped to sagaId + step name — replay-safe under at-least-once delivery.
• Saga orchestrator authZ reviewed — who can start, cancel, or force-compensate instances?
• Dashboards: running count, stuck sagas, compensation failure rate, p99 step latency per type.
• Client UX handles in-progress and failed-compensated terminal states without ambiguous "success".
• ADR captures choreography vs orchestration choice and rejected 2PC alternative.

Summary

The saga pattern enables multi-service business flows without distributed 2PC. Netflix production experience shows orchestrated sagas with idempotent steps, explicit compensations, and durable audit state contain blast radius when any leg fails — while keeping teams independently deployable.