Event Brokers

Introduction

Event brokers are the durable transport layer between producers and consumers — SNS/SQS, Amazon MSK, Google Pub/Sub, Azure Event Hubs. Staff architects choose brokers by delivery semantics (at-most-once, at-least-once, exactly-once), ordering guarantees, retention, and operational model — not by logo familiarity.

This lesson teaches broker selection matrices, multi-tenant topic design, dead-letter routing, and how Amazon's order pipeline survived a regional broker failover without losing checkout events.

Real production story

During Prime Day 2023, an Amazon retail team discovered their "temporary" RabbitMQ cluster still routed 40% of inventory events while MSK handled the rest. A misconfigured TTL on one exchange silently dropped price-update messages for 47 minutes — enough time for 12,000 SKUs to show stale prices on mobile. The post-incident review mandated a single broker platform per domain, explicit delivery semantics in every ADR, and DLQ dashboards tied to revenue SLOs.

Business problem

Business pressure: Amazon's marketplace must propagate price, inventory, and fulfillment events to 200+ downstream consumers within seconds. Without a deliberate broker strategy, teams adopt point-to-point queues, shared topics with no schema registry, and retry logic that amplifies load during incidents.

• Revenue at risk: Stale inventory or price events directly cause oversell and refund costs — measurable in millions per hour during peak.
• Engineering velocity: Each team wiring its own broker client library creates incompatible retry, DLQ, and observability patterns.
• Compliance / trust: Order and payment events require auditable delivery trails — "we think the message arrived" is not acceptable.

Architecture overview

An event broker accepts published messages, persists them according to retention policy, and delivers to subscribed consumers with defined guarantees. Production brokers are chosen per use case: SQS for work queues, SNS for fan-out, Kafka/MSK for event streaming with replay, RabbitMQ for complex routing.

• Definition: Durable middleware that implements publish-subscribe or point-to-point messaging with configurable delivery semantics.
• When to adopt: When more than three services need the same domain event, or when producers must not block on slow consumers.
• When to defer: Single producer, single consumer, low volume — a database outbox + polling worker may suffice.
• Operability: Broker lag, DLQ depth, consumer group offset, and publish error rate are first-class SLOs.

Architecture motivation

Why architects care: Event brokers decouple producers from consumers in time and topology. The naive alternative — synchronous HTTP fan-out from the order service — collapses under Prime Day traffic and couples every downstream team's deploy schedule to the order team's release cadence.

• Force: 1:N and N:M communication patterns with heterogeneous consumer speeds and availability.
• Constraint: Cannot re-platform all 200 consumers simultaneously; need bridge patterns during migration.
• Outcome: Standardized broker SDK with idempotency keys, schema validation, and DLQ triage runbooks per domain.

Internal architecture

Amazon marketplace event broker topology — domain-owned topics with platform guardrails:

• Producers never choose partition keys ad hoc — platform enforces tenant_id or order_id for ordering.
• Schema registry blocks incompatible publishes at the broker boundary, not in consumer catch blocks.
• DLQ is a first-class topic with its own consumer group and replay automation.

Order Service (producer)
   ↓ publish + schema_id
Schema Registry (Protobuf/Avro)
   ↓ validated payload
MSK Topic: orders.events.v2 (12 partitions, RF=3)
   ↓ consumer groups
├─ Inventory Service (group: inventory-v3)
├─ Fulfillment Service (group: fulfillment-v2)
├─ Analytics Lake (group: analytics-batch)
└─ DLQ: orders.events.v2.dlq → PagerDuty + replay tool

Data flow

Publish path: domain service writes to outbox table, relay process publishes to broker with idempotency key. Consume path: consumer commits offset only after side effects are durable.

• Write path: TX(order) + TX(outbox row) → relay → broker publish with message_id.
• Read path: Consumer polls/subscribes → validate schema → idempotent handler → ack offset.
• Async path: Failed messages route to DLQ after N retries with exponential backoff + jitter.

// Outbox relay + broker publish (TypeScript)
async function relayOutbox(batch: OutboxRow[]) {
  for (const row of batch) {
    await producer.send({
      topic: "orders.events.v2",
      key: row.aggregate_id,
      value: row.payload,
      headers: {
        "message-id": row.id,
        "schema-version": row.schema_version,
        "idempotency-key": row.idempotency_key,
      },
    });
    await db.outbox.markPublished(row.id);
  }
}

// Consumer with idempotency
async function handleOrderPlaced(event: OrderPlaced, ctx: ConsumeContext) {
  if (await idempotencyStore.seen(event.messageId)) {
    return ctx.commitOffset();
  }
  await inventoryService.reserve(event.orderId, event.items);
  await idempotencyStore.record(event.messageId);
  await ctx.commitOffset();
}

System design diagram

Two diagrams show the Event Brokers topology and the primary request/event path used in production at scale.

Production code example

Broker client wrapper — Amazon paved-road pattern with observability baked in:

• Wrap raw SDK calls — never scatter producer.send() across 40 services.
• Structured headers: message-id, trace-id, schema-version on every message.
• Metrics: publish success/error, consumer lag, DLQ depth per topic.

class BrokerClient {
  constructor(private producer: Producer, private metrics: Metrics) {}

  async publish(topic: string, event: DomainEvent): Promise<void> {
    const start = Date.now();
    try {
      await this.producer.send({
        topic,
        key: event.aggregateId,
        value: serialize(event),
        headers: { "message-id": event.id, "trace-id": trace.current() },
      });
      this.metrics.increment("broker.publish.success", { topic });
    } catch (err) {
      this.metrics.increment("broker.publish.error", { topic });
      throw new BrokerPublishError(topic, event.id, err);
    } finally {
      this.metrics.timing("broker.publish.latency", Date.now() - start, { topic });
    }
  }
}

Enterprise case study

Amazon — unified broker platform for retail domain: Platform team shipped a paved-road SDK wrapping MSK with schema registry, DLQ automation, and OpenTelemetry hooks. Product teams declare topics via self-service YAML; architecture review gates only P0 financial topics.

• Before: 14 broker technologies, no DLQ standards, 6-hour MTTR on message loss incidents.
• Decision: MSK as default; SQS for simple task queues; deprecate self-managed RabbitMQ by Q3.
• After: Broker lag visible per team; DLQ replay tool cut incident recovery from hours to 15 minutes.

Trade-offs

• Kafka vs SQS: Kafka offers replay and high throughput; SQS offers managed simplicity — choose per retention and ops appetite.
• Fan-out vs point-to-point: SNS+SQS fan-out adds latency and cost; direct queues simplify debugging but duplicate wiring.
• Ordering vs parallelism: Partition-key ordering limits horizontal scale — use only where business requires it.
• Managed vs self-hosted: MSK/Confluent Cloud trades control for ops burden; self-hosted Kafka needs dedicated SRE team.

Security considerations

Brokers are data planes: Order and payment events traverse them — encryption, ACLs, and audit logs are mandatory.

• Identity: IAM/SASL per producer and consumer principal — no shared cluster credentials.
• Data: Encrypt in transit (TLS) and at rest; scrub PII from event payloads where possible.
• Supply chain: Pin client SDK versions; scan broker images in CI before cluster upgrades.

Scalability analysis

Scale dimensions: Amazon brokers handle millions of events/minute during Prime Day. Partition count, consumer group parallelism, and broker disk I/O are the bottlenecks — not CPU on producers.

• Horizontal scale: Increase partitions before adding consumers; one consumer per partition max for ordered streams.
• Hot spots: A single hot partition key (e.g., one mega-seller) creates lag — salt keys or async aggregation.
• Cost: Retention × replication factor × partition count drives MSK bills — tier hot vs cold topics.

Failure scenarios

What breaks: Broker AZ failure, consumer poison messages, schema drift, and retry storms during partial outages.

• Broker partition leader election: Brief publish latency spike — producers need retry with idempotency enabled.
• Poison message: Consumer throws on every parse — route to DLQ after 3 attempts; never infinite retry.
• Schema incompatibility: New producer field breaks old consumer — enforce backward-compatible schema evolution.

Staff engineer insights

• Pick the broker by delivery semantics and retention needs — not because your last job used Kafka.
• DLQ depth is a product metric, not an ops afterthought. Alert on it before customers notice stale data.
• Every consumer team owns their idempotency store — the broker guarantees at-least-once, not exactly-once processing.
• Schema registry at the broker boundary saves more incidents than any consumer-side try/catch.

Interview questions

Architecture review questions

• Delivery semantics documented per topic — at-least-once with idempotent consumers?
• Schema registry enforces backward-compatible evolution?
• DLQ exists, is monitored, and has a replay runbook?
• Partition key strategy documented — no accidental hot partitions?
• IAM/SASL credentials scoped per producer/consumer principal?
• Broker lag and publish error rate on team dashboards with SLO alerts?

Summary

Event brokers decouple producers and consumers at scale. At Amazon-class traffic, success depends on delivery semantics, schema governance, DLQ operations, and idempotent consumers — not broker brand. Choose by forces, wire observability from day one, and treat DLQ depth as a revenue signal.