Orchestration

Introduction

Orchestration coordinates multi-step distributed workflows from a central process manager that invokes services and tracks state. Airbnb's booking saga (reserve → charge → confirm → notify host) uses orchestration because compensation paths are complex and regulators expect a single auditable workflow record.

Staff architects choose orchestration when saga compensation, ordering guarantees, and workflow visibility outweigh the coupling cost of a central coordinator.

Real production story

Airbnb's booking flow requires: hold inventory, pre-authorize payment, confirm with host, and release hold on failure — four services with strict compensation (release hold, void charge, cancel notification). Early choreography caused double-bookings when payment succeeded but inventory release event was lost. Moving to an orchestrated saga with a durable workflow engine (Temporal) gave a single state machine, automatic retries per step, and a queryable workflow history that satisfied SOX audit requirements for payment flows.

Business problem

Business pressure: Airbnb bookings involve money, inventory, and host commitments — partial completion is a customer trust and regulatory incident. "We think the booking completed" is not acceptable for SOX-scoped payment flows.

• Revenue at risk: Double-bookings and orphaned charges drive refunds, chargebacks, and host churn.
• Engineering velocity: Choreographed compensation bugs require cross-team war rooms — orchestration centralizes failure handling.
• Compliance / trust: Auditors need single workflow history showing each step, timestamp, and compensation action.

Architecture overview

Orchestration uses a central coordinator (workflow engine) that calls participant services, records state transitions, executes compensations on failure, and exposes workflow status queries.

• Definition: Central process manager driving saga steps with durable state and compensation logic.
• When to adopt: Complex compensation, long-running workflows, audit requirements, strict step ordering.
• When to defer: Simple event reactions with no compensation — choreography is simpler.
• Operability: Workflow completion rate, step latency, compensation frequency, and stuck-workflow alerts.

Architecture motivation

Why architects care: Orchestration provides a durable state machine with explicit compensation transitions. The naive choreographed alternative scattered compensation across four services — any missed compensating event left money captured without inventory released.

• Force: Multi-step workflow with compensating transactions and regulatory audit trail.
• Constraint: Steps have different timeouts (payment 30s, host confirm 24h) — need durable timers.
• Outcome: Temporal/Conductor workflow with versioned saga definitions and queryable history API.

Internal architecture

Airbnb booking saga orchestration — Temporal workflow:

• Each activity has explicit timeout and retry policy — not buried in consumer code.
• Compensation sequence defined in workflow — reverse order of successful steps.
• Durable timers survive worker restarts — 24h host wait is not a sleeping thread.

Booking API → StartWorkflow(BookingSaga, booking_id)

Temporal: BookingSaga workflow
  1. Activity: InventoryService.reserve(listing_id, dates)
     → on fail: END (no compensation needed)
  2. Activity: PaymentService.preAuthorize(guest_id, amount)
     → on fail: compensate InventoryService.release()
  3. Timer: wait host confirmation (24h)
     → Activity: HostService.confirm(booking_id)
     → on fail/timeout: compensate Payment.void() + Inventory.release()
  4. Activity: NotificationService.sendConfirmation()
     → on fail: retry 3x (non-critical, no compensation)

Workflow history: queryable via booking_id for audit

Data flow

Start: API starts workflow with booking_id as workflow ID (idempotent). Execute: engine calls activities sequentially/parallel per definition. Fail: engine runs compensation stack in reverse order of completed steps.

• Write path: Workflow start → activity calls with idempotency keys → state persisted after each step.
• Read path: Query workflow status API returns current step + history for support and audit.
• Async path: Durable timer fires host-timeout → triggers compensation without polling.

// Temporal booking saga (TypeScript)
export async function bookingSaga(input: BookingInput): Promise<BookingResult> {
  let reservationId: string | undefined;
  let chargeId: string | undefined;

  try {
    reservationId = await activities.reserveInventory(input.listingId, input.dates);
    chargeId = await activities.preAuthorizePayment(input.guestId, input.amount);

    const confirmed = await condition(() => hostConfirmed(input.bookingId), "24h");
    if (!confirmed) throw new Error("Host confirmation timeout");

    await activities.sendConfirmation(input.bookingId);
    return { status: "CONFIRMED", bookingId: input.bookingId };
  } catch (err) {
    if (chargeId) await activities.voidPayment(chargeId);
    if (reservationId) await activities.releaseInventory(reservationId);
    await activities.notifyGuestBookingFailed(input.guestId, input.bookingId);
    return { status: "FAILED", reason: err.message };
  }
}

// Activity with idempotency
activities.reserveInventory = proxyActivities<typeof acts>({
  startToCloseTimeout: "30s",
  retry: { maximumAttempts: 3 },
});

System design diagram

Two diagrams show the Orchestration topology and the primary request/event path used in production at scale.

Production code example

Temporal saga deployment — Airbnb production pattern:

• Separate task queues per saga version for safe blue-green workflow deploys.
• Activity timeouts tuned per external dependency SLA — not one global timeout.
• Alert on workflows exceeding expected duration per step.

// Worker deployment
const worker = await Worker.create({
  connection: await Connection.connect({ address: temporal.prod.airbnb.com }),
  namespace: "bookings",
  taskQueue: "booking-saga-v3",
  workflowsPath: require.resolve("./workflows"),
  activities: { ...inventoryActs, ...paymentActs, ...notificationActs },
  maxConcurrentActivityExecutions: 200,
});

// Stuck workflow alert
// booking_saga_running > 25h AND step = "awaiting_host_confirm"
alert "stuck_booking_saga" {
  expr = 'temporal_workflow_running{workflow="BookingSaga"} > 0'
  for  = "25h"
  labels = { severity = "page" }
}

Enterprise case study

Airbnb — booking saga on Temporal: Central orchestration for payment-scoped booking flow with queryable audit history. Choreography retained for non-financial events (search indexing, recommendation updates).

• Before: Choreographed booking caused 3 double-booking incidents per quarter; audit required manual log correlation.
• Decision: Temporal for booking saga; event catalog for non-financial downstream reactions.
• After: Zero double-bookings in 18 months; audit pulls workflow history by booking_id in seconds.

Trade-offs

• Visibility vs coupling: Orchestration gives single workflow view but centralizes dependency on workflow engine.
• Engine SPOF: Temporal/Conductor cluster must be HA — but workers scale horizontally.
• Versioning: Workflow code changes need versioning strategy — breaking in-flight workflows is an incident.
• Choreography alternative: Simpler for loose coupling but compensation is error-prone across services.

Security considerations

Workflow history contains PII and payment data: Encrypt at rest, RBAC on query API, audit log on history access.

• Identity: Activity calls use service-to-service mTLS; workflow engine admin API behind SSO.
• Data: Minimize PII in workflow input — reference IDs, not full guest profiles.
• Supply chain: Pin workflow SDK version; test saga changes in shadow environment with production traffic replay.

Scalability analysis

Scale dimensions: Airbnb processes millions of bookings. Workflow engine handles state persistence; activity workers scale per step type.

• Horizontal scale: Payment activity workers scale independently from notification workers.
• Hot spots: Holiday booking spikes — rate-limit workflow starts per guest_id to prevent abuse.
• Cost: Workflow history storage grows with completed bookings — retention policy per compliance needs.

Failure scenarios

What breaks: Activity succeeds but workflow crashes before recording; compensation activity fails; workflow version mismatch on deploy.

• Lost acknowledgment: Payment charged but workflow timed out — idempotent activity + workflow ID = booking_id prevents double charge.
• Compensation failure: voidPayment fails — alert + manual intervention queue; never silently skip.
• Version deploy: In-flight workflows on old version — use Temporal workflow versioning or drain before deploy.

Staff engineer insights

• Orchestrate when compensation is harder than coordination — Airbnb bookings, not Uber trip logging.
• Workflow ID = business idempotency key — booking_id as workflow ID prevents duplicate sagas.
• Compensation activities need the same idempotency rigor as forward activities.
• Version workflow code like database migrations — never break in-flight executions.

Interview questions

Architecture review questions

• Compensation sequence defined for every failure point after partial success?
• Workflow ID = business idempotency key?
• Activity timeouts and retry policies per external dependency?
• Workflow versioning strategy for deploys documented?
• Stuck-workflow alerts per step with expected duration?
• Audit query API returns complete history for compliance scope?

Summary

Orchestration at Airbnb scale means durable sagas with explicit compensation, workflow IDs as idempotency keys, and queryable history for audit. Choose orchestration when compensation and compliance matter more than maximum decoupling — and operate the workflow engine as critical infrastructure.