Outbox Pattern

Introduction

Outbox Pattern solves the dual-write problem: atomically persist business data and outgoing message in the same database transaction, then a relay process publishes to Kafka/SNS. No lost events when broker is down after DB commit.

The story

Place order wrote to DB then published to Kafka in same handler. Kafka down after DB commit — 4,000 missing events after outage. Outbox: events in outbox table same transaction; poller relays to Kafka. Zero lost events since.

The business problem

Separate DB commit and message publish causes silent data loss:

• Dual-write: DB succeeds, broker fails — downstream never notified.
• In-memory queue: crash after commit loses unpublished events.
• No XA: distributed TX across DB + Kafka too slow/unavailable.
• Silent downstream drift: missing events corrupt read models.

The problem teams faced

Outbox fits when:

• Must reliably publish event after DB state change.
• At-least-once delivery to message broker required.
• Can't use Kafka transactions tied to DB in your stack.
• Event-driven projections downstream of write DB.

Understanding the topic

Intent: Atomic write of business data + outbox row; async relay to broker.

• Outbox table — event payload, type, created_at, published flag.
• Business transaction — INSERT order + INSERT outbox in same TX.
• Relay/poller — reads unpublished outbox rows, publishes, marks published.
• Idempotent consumers — at-least-once delivery downstream.

Internal architecture

Outbox in place order handler:

async function placeOrder(cmd: PlaceOrder) {
  await db.transaction(async (tx) => {
    const order = await tx.orders.insert(cmd)
    await tx.outbox.insert({
      aggregateId: order.id,
      type: "OrderPlaced",
      payload: JSON.stringify(order.toEvent()),
      published: false,
    })
  })
}

// Separate relay worker
async function relayOutbox() {
  const rows = await db.outbox.where({ published: false }).limit(100)
  for (const row of rows) {
    await kafka.publish(row.type, row.payload)
    await db.outbox.update(row.id, { published: true })
  }
}

Visual explanation

Three diagrams cover dual-write problem, outbox flow, and relay:

Informative example

Before / after — dual write to outbox:

// ❌ Dual write — event lost if Kafka down
async function placeOrder(cmd) {
  await db.orders.insert(cmd)
  await kafka.publish("OrderPlaced", cmd)  // fails after insert committed
}

// ✅ Outbox — atomic
async function placeOrder(cmd) {
  await db.transaction(async (tx) => {
    await tx.orders.insert(cmd)
    await tx.outbox.insert({ type: "OrderPlaced", payload: cmd, published: false })
  })
}
// Relay publishes later — survives broker outage

Execution workflow

Real-world use

Debezium CDC outbox pattern, Netflix, Uber outbox relays, transactional outbox in microservices.io pattern catalog, MassTransit EF outbox.

Production case study

4,000 missing Kafka events fix:

• Symptom: DB committed, Kafka publish failed — 4,000 events lost.
• Decision: Outbox table + relay poller.
• Outcome: zero lost events since deployment.

Trade-offs

• Pro: reliable publish; survives broker outages; same-DB atomicity.
• Con: eventual publish lag (poll interval).
• Con: relay ops — monitor lag, duplicate handling downstream.

Decision framework

• Use when publishing events after DB write in microservices.
• Use with Event Sourcing + CQRS projections fed by broker.
• CDC (Debezium) vs poller — CDC lower lag, more infra.
• Skip when fire-and-forget OK and loss tolerable — rare in prod.

Best practices

• Never publish directly in handler after DB write — outbox always.
• Monitor outbox lag — alert on unpublished row age.
• Consumer idempotency keys — event ID dedup.
• Cleanup published rows — retention policy.

Anti-patterns to avoid

• In-memory event queue after commit — lost on crash.
• XA transactions DB+Kafka — operational pain, often avoided.
• Outbox without relay monitoring — silent publish stall.

Common mistakes

• Relay marks published before Kafka ack — event still lost — ack then mark.
• Duplicate publish on relay retry — consumers must dedupe.

Debugging tips

• Count unpublished outbox rows — backlog indicates relay/broker issue.
• Compare outbox payload to consumed message — serialization bugs.

Optimization strategies

• Debezium CDC for sub-second relay vs polling.
• Batch publish from relay for throughput.

Common misconceptions

• Outbox ≠ Event Sourcing. Outbox is reliable publish mechanism; ES is storage model. Often combined.
• Exactly-once end-to-end rare — outbox gives at-least-once; idempotent consumers complete story.

Advanced interview questions

Summary

You can implement outbox table, relay worker, and explain dual-write fix. Tell the 4,000 missing events story — if you can do that, you own Outbox.