Event-Driven Architecture

Introduction

Event-Driven Architecture (EDA) replaces synchronous "call and hope" chains with immutable facts published to a log. Producers emit what happened — OrderPlaced, PaymentCaptured — and consumers react independently. The order service doesn't know who listens; new subscribers appear without changing the publisher. At scale, that's how checkout decouples from email, search indexing, fraud scoring, and analytics without blocking the user.

The story

A marketplace checkout path synchronously HTTP-called inventory, billing, email, analytics, and search — five hops in the critical path. When the email provider slowed to 8s timeouts, checkout p99 hit 12s and conversion dropped 14%. Rebuild: order service writes DB + outbox row in one transaction; relay publishes OrderPlaced to Kafka. Five consumers fan out async. Checkout p99 fell to 380ms; email outages no longer touch revenue.

The business problem

Synchronous orchestration in growing systems creates coupling finance and on-call feel directly:

• Critical-path bloat: slowest downstream sets user-facing latency for everyone.
• Blast radius: one dead partner API blocks unrelated features in the same request chain.
• Scaling mismatch: email/analytics need different throughput than order writes — sync forces same SLA.
• Audit gaps: HTTP fire-and-forget leaves no durable record of "what we told the business happened."
• Feature drag: adding a subscriber means editing and redeploying the publisher's orchestration code.

The problem teams faced

Event-Driven Architecture fits when:

• Multiple downstream systems must react to the same business fact independently.
• Side effects (email, indexing, analytics) can be asynchronous without blocking the user.
• You need an audit trail of state changes across services.
• Teams want to add consumers without redeploying producers.
• Peak write traffic must not be limited by slowest read-side processor.

Understanding the topic

Intent: Propagate state changes as durable events; decouple producers from consumers via a message backbone.

• Domain event — past-tense fact: OrderPlaced, not PlaceOrder (command).
• Event bus / log — Kafka, Pulsar, SNS+SQS — durable, replayable fan-out.
• Producer — appends event after successful state change (often via Outbox).
• Consumer — idempotent handler; at-least-once delivery assumed.
• Schema registry — Avro/Protobuf/JSON Schema for compatible evolution.

Internal architecture

Production EDA stack — order placement:

Client → API Gateway → Order Service
                    │
         ┌──────────┴──────────┐
         ▼                     ▼
    orders table          outbox table
    (same DB TX)          (same DB TX)
         │                     │
         │              Relay Worker → Kafka: orders.events
         │                     │
         │         ┌───────────┼───────────┬───────────┐
         │         ▼           ▼           ▼           ▼
         │    Inventory   Email      Analytics   Search
         │    consumer    consumer   consumer    indexer
         ▼
    202 + orderId (user response — doesn't wait for consumers)

Visual explanation

Three diagrams cover sync vs async checkout, outbox + fan-out, and event taxonomy:

Informative example

Before / after — sync orchestration to outbox + Kafka:

// ❌ Sync chain — email slowness blocks checkout
async function placeOrder(cmd: PlaceOrderCommand) {
  const order = await db.orders.insert(cmd)
  await inventory.reserve(order)      // sync
  await payment.charge(order)         // sync
  await email.sendReceipt(order)        // 8s timeout kills p99
  await analytics.track(order)          // sync
  await search.index(order)             // sync
  return order
}

// ✅ Event-driven — user path ends after DB + outbox
async function placeOrder(cmd: PlaceOrderCommand) {
  const order = Order.create(cmd)
  await db.transaction(async (tx) => {
    await tx.orders.insert(order)
    await tx.outbox.insert({
      eventId: uuid(),
      type: "OrderPlaced",
      payload: order.toEvent(),
    })
  })
  return order  // relay publishes async; consumers react independently
}

// Consumer — idempotent
async function onOrderPlaced(event: OrderPlaced) {
  if (await processed.has(event.eventId)) return
  await email.sendReceipt(event)
  await processed.mark(event.eventId)
}

Execution workflow

Real-world use

Amazon event-driven microservices; Uber Kafka backbone; Netflix Keystone pipeline; Stripe webhook + internal event bus; LinkedIn Kafka trillions/day; Debezium CDC as alternative to application outbox for DB-change events.

Production case study

Marketplace checkout decoupling:

• Before: 5 sync HTTP calls; email timeout → checkout p99 12s; conversion −14%.
• Decision: OrderPlaced to Kafka via outbox; five independent consumers.
• Outcome: checkout p99 380ms; email outages isolated from revenue path.
• Ops: consumer lag dashboard + DLQ for poison messages.

Trade-offs

• Pro: loose coupling; independent scaling; audit log; add consumers without publisher changes.
• Pro: absorbs traffic spikes — queue buffers slow consumers.
• Con: eventual consistency — user may not see side effect immediately.
• Con: operational complexity — lag, DLQ, schema migration, distributed debug.
• Con: harder reasoning — "what state is the system in right now?" needs projections.

Decision framework

• Use EDA when ≥2 independent reactions to same fact and user doesn't need them sync.
• Keep payment/inventory reservation sync or saga-orchestrated — not fire-and-forget event.
• Always outbox (or CDC) — never publish-then-write or write-then-naked-publish.
• Start with one topic per bounded context — not one event bus god-topic.
• Skip Kafka for 2-person startup with 3 events/day — Postgres NOTIFY or simple queue suffices.

Best practices

• Events past tense: OrderPlaced, PaymentFailed — not PlaceOrder.
• Include eventId, type, version, occurredAt, correlationId in envelope.
• Schema registry + backward-compatible evolution (add optional fields).
• Dead letter queue after N retries; alert on DLQ depth.
• Consumer idempotency store (Redis/DB) keyed by eventId.
• Trace correlationId from HTTP request through all consumers.

Anti-patterns to avoid

• Chatty events — 50 micro-events per user click; batch or use domain-meaningful granularity.
• Event-as-API — synchronous request/response disguised as events with blocking reply topic.
• Dual write without outbox — DB commits, Kafka publish fails, state diverges forever.
• God consumer — one service handles every event type; becomes new monolith.
• Breaking schema changes without version — consumers crash on deploy.

Common mistakes

• Ordering assumptions across partitions — Kafka only orders per partition key.
• Using userId as partition key then wondering why one celebrity hot-spots shard.
• No replay strategy — bug fixed in consumer but bad events already processed.
• Circular events — A emits → B emits → A loops infinitely.

Debugging tips

• Follow correlationId across producer TX, relay, Kafka, each consumer span.
• Graph consumer lag per group — which handler fell behind?
• Replay from offset to staging consumer before reprocessing production DLQ.

Optimization strategies

• Partition by orderId (not userId) if order events must stay ordered per order.
• Compacted topics for changelog-style state (Kafka compaction).
• Batch consumer polls for high-throughput analytics handlers.

Common misconceptions

• EDA ≠ no orchestration. Sagas still coordinate multi-step workflows — events carry facts between steps.
• EDA ≠ Event Sourcing. EDA is communication style; ES stores events as source of truth. Often combined, not same thing.
• Kafka ≠ required. SQS+SNS, RabbitMQ, NATS fit smaller scale — same principles, different ops profile.
• At-least-once ≠ bad. Industry default; idempotent consumers make it correct.

Advanced interview questions

Summary

You can draw sync vs event-driven checkout, explain outbox + Kafka fan-out, and defend when NOT to use EDA. Tell the marketplace p99 story — if you can do that, you own Event-Driven Architecture.