Outbox Pattern

Introduction

Transactional outbox guarantees that domain state changes and outbound messages commit atomically in one local database transaction. A relay process publishes rows from the outbox table to Kafka or SQS, eliminating dual-write races that plague "update DB then publish event" designs. Uber's dispatch and billing pipelines rely on this pattern at millions of events per minute.

Real production story

Uber's trip pricing service updated PostgreSQL fare totals and separately called a Kafka producer. Under GC pauses, some trips persisted final fares but never emitted TripCompleted — downstream driver payouts and city tax reports drifted. Reconciliation jobs found 12k mismatched trips per week in one region.

Engineers introduced an outbox table written in the same transaction as fare updates. Debezium CDC (and a fallback poller) streamed outbox rows to Kafka with ordering per tripId. Dual-write anomalies fell below measurable threshold; payout SLOs recovered without bringing pricing and messaging into one deploy unit.

Business problem

Business pressure: Uber's marketplace requires fare, trip state, and payout events to agree within seconds across regions. Missing events mean drivers unpaid and regulators receiving incomplete trip records.

• Financial accuracy: Event loss directly translates to incorrect driver earnings and tax remittance.
• Operational load: Weekly reconciliation teams are expensive and do not scale with trip volume.
• Decoupling goal: Teams want database-per-service without giving up reliable cross-service notification.

Architecture overview

Outbox = table in the same database as domain data. Application inserts domain row + outbox row in one transaction. Relay = CDC (Debezium) or polling publisher marks rows published and pushes to broker.

• Definition: Transactional message staging inside the service's authoritative store.
• When to adopt: Any time domain write must reliably trigger downstream async processing.
• When to defer: Fire-and-forget analytics where loss is acceptable — use direct publish.
• Operability: Monitor outbox lag (unpublished row age) as a first-class SLO.

Architecture motivation

Why architects care: The outbox makes "write state" and "notify others" one atomic unit locally; message brokers remain eventually consistent globally — a provable improvement over best-effort dual writes.

• Force: At-least-once brokers + crash between DB commit and publish = silent data loss.
• Constraint: Cannot block trip completion on downstream consumer health.
• Outcome: Single local TX boundary; relay handles retry, ordering keys, and poison messages.

Internal architecture

Uber trip-fare outbox topology:

TripPricingService
        │
        ├─ BEGIN TX
        │    UPDATE trips SET fare = ?
        │    INSERT INTO outbox(id, aggregate_id, type, payload, created_at)
        │  COMMIT
        │
        ↓ (WAL / logical replication)
   Debezium Connector → Kafka topic: trip.events
        │
        ├─ PayoutService (consumer)
        ├─ TaxReportingService (consumer)
        └─ Analytics (consumer)

Fallback: OutboxPoller (cron) if CDC lag > threshold
Metrics: outbox_unpublished_count, relay_lag_seconds

Data flow

Write path is synchronous-local; fan-out is async-global.

• Write path: Domain mutation + outbox insert in single DB transaction — never call Kafka inside TX.
• Read path: Consumers read from Kafka; authoritative fare still in trips table.
• Relay path: CDC reads WAL; maps outbox row to CloudEvent; commits offset after broker ack.

BEGIN;
  UPDATE trips SET status='COMPLETED', fare_cents=1250 WHERE id=$1;
  INSERT INTO outbox (id, aggregate_id, event_type, payload)
  VALUES (gen_random_uuid(), $1, 'TripCompleted', $2);
COMMIT;

System design diagram

Two diagrams show the Outbox Pattern topology and the primary request/event path used in production at scale.

Production code example

Application-side outbox write with relay idempotency:

@Transactional
public Trip finalizeTrip(UUID tripId, Fare fare) {
  Trip trip = tripRepo.findForUpdate(tripId);
  trip.complete(fare);
  tripRepo.save(trip);

  OutboxEvent event = OutboxEvent.builder()
      .id(UUID.randomUUID())
      .aggregateId(tripId.toString())
      .eventType("TripCompleted")
      .payload(objectMapper.writeValueAsString(TripCompleted.of(trip)))
      .createdAt(Instant.now())
      .published(false)
      .build();
  outboxRepo.save(event);
  return trip;
}

// Relay (separate process)
@Scheduled(fixedDelay = 100)
void publishPending() {
  List<OutboxEvent> batch = outboxRepo.findTop500ByPublishedFalseOrderByCreatedAt();
  for (OutboxEvent e : batch) {
    kafkaTemplate.send("trip.events", e.getAggregateId(), e.getPayload());
    outboxRepo.markPublished(e.getId());
  }
}

Enterprise case study

Uber trip completion → payout pipeline migrated from dual-write to transactional outbox + Debezium.

• Before: 12k trip/event mismatches per week in one region; manual payout adjustments.
• Decision: Outbox table + CDC with poller fallback; TripCompleted keyed by tripId.
• After: Mismatch rate below 0.001%; outbox lag p99 under 800ms at peak.

Trade-offs

• Latency vs correctness: Events appear after relay lag (ms–s), not inline with HTTP response.
• CDC vs polling: CDC is lower latency but operationally heavier; polling simpler but adds DB read load.
• Schema coupling: Outbox payload format is a contract — versioning and compatibility rules required.
• Ordering: Global order impossible; partition by aggregate_id for per-trip ordering.

Security considerations

Outbox payloads often contain PII and fare data — treat relay path as sensitive.

• Encryption: TLS to Kafka; encrypt payload at rest if topic is multi-tenant.
• Access control: Only relay service account reads outbox table; app role INSERT-only.
• Audit: Immutable outbox IDs correlate to tripId for fraud investigations.

Scalability analysis

Outbox tables grow fast at Uber trip volume — archival and indexed polling matter.

• Horizontal scale: Relay consumers scale with Kafka partitions keyed by tripId.
• Hot spots: Mega-events (NYE) spike outbox inserts — batch relay and autoscale connectors.
• Cost: Retain published rows 7 days then purge; cold archive for audit if required.

Failure scenarios

Relay failures must not lose rows — unpublished outbox entries are the source of truth.

• CDC connector crash: Kafka Connect restarts from last offset; duplicates handled by consumer idempotency.
• Broker unavailable: Outbox rows accumulate; alert on age p99 > 30s; backpressure trip API if needed.
• Poison payload: DLQ after N relay attempts; domain TX already committed — fix forward with patch event.

Staff engineer insights

• Never publish to Kafka inside a database transaction — you have already lost atomicity.
• Outbox lag is a product SLO: when relay falls behind, downstream UX degrades before DB does.
• Design outbox schema for evolution: event_type + version + opaque payload, not one JSON blob per release.

Interview questions

Architecture review questions

• Domain write and outbox insert share one transaction boundary — verified in code review.
• Relay lag monitored with alert threshold tied to business SLO (payout delay).
• Outbox retention and purge job tested — table growth modeled at 10× traffic.
• Event schema versioned; consumers tolerate unknown fields.
• Fallback poller exists when CDC is down — runbook documents switchover.
• Partition key = aggregate id for ordering guarantees documented per event type.

Summary

The transactional outbox pattern gives Uber-class systems atomic "state + notify" locally while keeping microservices decoupled. CDC or polling relays bridge to Kafka reliably — turning reconciliation fire drills into rare edge cases.