Choreography

Introduction

Choreography coordinates distributed workflows through events alone — no central orchestrator. Each service reacts to domain events and publishes its own. Uber's trip lifecycle (request → match → pickup → complete → pay) evolved from orchestrated dispatch to choreographed events as service count grew.

Staff architects use choreography when services are loosely coupled, workflows are discoverable via event catalog, and central state machines become bottlenecks.

Real production story

Uber's trip completion flow originally ran through a central dispatch orchestrator that called payment, rating, and receipt services synchronously. When payment service degraded, the orchestrator's thread pool exhausted and blocked new trip matches — a correlated failure across unrelated domains. Migrating to choreographed events (TripCompleted → PaymentService, RatingService, ReceiptService each subscribe independently) isolated blast radius. The trade-off surfaced three months later: no single view of "trip close workflow" made debugging a stuck payout require correlating events across five services.

Business problem

Business pressure: Uber's trip lifecycle spans matching, routing, payment, and safety services across regions. A central orchestrator becomes a scaling bottleneck and single point of failure during city-scale events (concerts, storms).

• Revenue at risk: Orchestrator saturation blocks new trip requests — direct revenue loss per minute in peak cities.
• Engineering velocity: Every new trip-lifecycle step requires orchestrator code change and deploy — serializes teams.
• Compliance / trust: Payment and safety events need traceable causality chains without a fragile central coordinator.

Architecture overview

Choreography means each service knows only: which events it consumes and which events it publishes. No service knows the full workflow graph. Coordination emerges from event chains.

• Definition: Decentralized workflow coordination via publish-subscribe with no central process manager.
• When to adopt: Loosely coupled domains, high service count, teams own full vertical slices.
• When to defer: Strict ordering across many steps, complex compensation, or regulatory need for central audit trail.
• Operability: Distributed tracing, event catalog, and correlation_id on every event are mandatory.

Architecture motivation

Why architects care: Choreography decouples services in time and ownership. Each team owns its reaction to TripCompleted without negotiating with a central workflow team. The cost is distributed observability and compensating logic without a central saga coordinator.

• Force: 10+ services react to same domain event with independent deploy cadences.
• Constraint: Cannot afford orchestrator as SPOF during Super Bowl-scale demand spikes.
• Outcome: Event catalog, correlation IDs, and distributed tracing as mandatory platform capabilities.

Internal architecture

Uber trip close choreography — no central orchestrator:

• correlation_id = trip_id propagated on every event for distributed trace stitching.
• Each consumer group scales independently — payment lag does not block rating.
• Event catalog documents: TripCompleted triggers 4 consumers — discoverability replaces orchestrator graph.

Trip Service
   ↓ publishes TripCompleted { trip_id, rider_id, fare, correlation_id }

Event Bus (Kafka: trips.lifecycle.v1)
   ↓ independent consumer groups
├─ Payment Service
│    consumes TripCompleted → charges rider → publishes PaymentCompleted
├─ Rating Service
│    consumes TripCompleted → prompts rating → publishes RatingRequested
├─ Receipt Service
│    consumes PaymentCompleted → generates PDF → publishes ReceiptSent
└─ Safety Service
     consumes TripCompleted → logs trip summary → publishes SafetyLogWritten

No service calls another directly — all via events + correlation_id

Data flow

Trigger: Trip service publishes TripCompleted after DB commit (outbox). Reactions: each subscriber processes independently, publishes downstream events. No rollback coordinator — compensation via compensating events (PaymentFailed → RefundInitiated).

• Write path: Trip state → COMPLETED in DB → outbox → TripCompleted event.
• Read path: Each service maintains own projection; no shared workflow state table.
• Async path: PaymentFailed → PaymentService publishes → RefundService and NotificationService react.

// Trip service — publish after local commit
async function completeTrip(tripId: string) {
  await db.transaction(async (tx) => {
    await tx.trips.update(tripId, { status: "COMPLETED", completedAt: new Date() });
    await tx.outbox.insert({
      eventType: "TripCompleted",
      aggregateId: tripId,
      payload: { tripId, riderId, fare, correlationId: tripId },
    });
  });
}

// Payment service — independent choreographed reaction
async function onTripCompleted(event: TripCompleted) {
  try {
    const charge = await stripe.charge(event.riderId, event.fare);
    await publish("PaymentCompleted", { tripId: event.tripId, chargeId: charge.id, correlationId: event.correlationId });
  } catch (err) {
    await publish("PaymentFailed", { tripId: event.tripId, reason: err.message, correlationId: event.correlationId });
  }
}

System design diagram

Two diagrams show the Choreography topology and the primary request/event path used in production at scale.

Production code example

Event catalog + correlation tracing — Uber platform pattern:

• Event catalog is the choreography diagram — producers and consumers declared explicitly.
• Tracing middleware stamps correlation_id on every span for stuck-workflow detection.
• Reconciliation cron finds correlation_ids with TripCompleted but no PaymentCompleted after 10 min.

// Event catalog entry (YAML)
event: TripCompleted
version: 2
producer: trip-service
consumers: [payment-service, rating-service, safety-service]
schema: trips/TripCompleted.avsc
sla:
  publishLatencyP99Ms: 500

// Tracing middleware on every consumer
function withCorrelation(handler: EventHandler): EventHandler {
  return async (event, ctx) => {
    const span = tracer.startSpan("consume.TripCompleted", {
      childOf: extractTraceContext(event.headers),
      tags: { correlation_id: event.correlationId, trip_id: event.tripId },
    });
    try {
      await handler(event, ctx);
    } finally {
      span.finish();
    }
  };
}

Enterprise case study

Uber — choreographed trip lifecycle: Migrated payment, rating, receipt from central orchestrator to independent event consumers. Invested in correlation_id tracing and event catalog before cutover.

• Before: Orchestrator SPOF caused correlated failures; new lifecycle steps required central team deploy.
• Decision: Choreography for trip close; keep orchestration only for real-time matching (latency-critical).
• After: Payment degradation no longer blocks trip matching; MTTR improved via distributed trace by correlation_id.

Trade-offs

• Coupling vs visibility: Choreography reduces coupling but hides global workflow — need event catalog and tracing.
• Compensation: No central saga — compensating events must be designed per failure path; risk of incomplete compensation.
• Ordering: Events may arrive out of order — consumers must handle with idempotency and version checks.
• Debugging: "Stuck workflow" requires correlating events across services — invest in observability upfront.

Security considerations

Distributed events carry financial data: Each hop needs auth, encryption, and minimal payload exposure.

• Identity: Each consumer validates event signature or mTLS source — no trust-by-topic alone.
• Data: TripCompleted carries fare, not full payment instrument — tokenized references only.
• Supply chain: Event schema changes reviewed for PII field additions.

Scalability analysis

Scale dimensions: Uber completes millions of trips daily. Choreography scales because each consumer group scales independently — no orchestrator thread pool ceiling.

• Horizontal scale: Add payment consumers without touching rating or receipt services.
• Hot spots: City-wide surge creates TripCompleted burst — partition by city_id for geographic isolation.
• Cost: More topics and consumer groups than orchestration — offset by eliminated orchestrator SPOF infra.

Failure scenarios

What breaks: Payment succeeds but receipt never generates; duplicate TripCompleted causes double charge; missing compensating event leaves inconsistent state.

• Partial completion: PaymentCompleted lost — receipt waits forever — timeout + alert on correlation_id stuck > 5 min.
• Duplicate event: At-least-once delivery double-charges — idempotency key on payment handler.
• Missing compensation: PaymentFailed published but no RefundInitiated — periodic reconciliation job detects orphan failures.

Staff engineer insights

• Choreography is not "no coordination" — it moves coordination to event contracts and observability.
• If you cannot draw the workflow from the event catalog, you are not ready to choreograph.
• Compensating events are your distributed rollback — design them before the first production failure.
• correlation_id on every event is non-negotiable — without it, debugging is archaeology.

Interview questions

Architecture review questions

• Event catalog documents all producers, consumers, and schemas?
• correlation_id propagated on every event in the workflow?
• Compensating events designed for each failure path?
• Idempotency on all choreographed consumers?
• Reconciliation job for stuck correlation_ids?
• Distributed tracing spans linked by correlation_id?

Summary

Choreography at Uber scale means independent services reacting to domain events with correlation tracing, event catalogs, and compensating events — not a central orchestrator. The pattern scales teams and traffic; observability is the price of admission.