Distributed Systems Patterns

Introduction

Distributed patterns are the staff engineer's toolkit for building correct systems when a single @Transactional boundary no longer spans the work. In monolithic Java, one database and one transaction guaranteed ACID. In microservices, an order spans inventory, payment, and shipping — each with its own Postgres — and network failure is guaranteed, not exceptional.

This lesson covers five patterns every Java architect must implement or review:

Saga — coordinate multi-service transactions via compensating actions. CQRS — separate read and write models for scale and clarity. Event Sourcing — persist state as immutable events, rebuild anytime. Outbox — reliably publish events without dual-write corruption. Circuit Breaker — fail fast when dependencies unhealthy, prevent cascade.

Spring Boot teams use these daily — with Resilience4j, Spring Modulith events, and Debezium outbox. Real-world case studies from Uber, Shopify, and LinkedIn show why patterns beat heroic distributed transactions.

Business problem

Without distributed patterns, microservices become distributed monoliths with worse failure modes:

• Dual-write corruption: Update Postgres then publish to Kafka — crash between steps → inconsistent state; customer charged but order not created.
• 2PC/XA avoided then reinvented badly: Teams try synchronous REST chains with no compensation — payment succeeds, inventory fails, money trapped.
• Read model overload: Complex JOIN queries on write-optimized schema — catalog page requires 8 service calls, p99 4 seconds.
• Cascade failure: Slow fraud service exhausts checkout thread pool — entire platform down because one dependency lacked circuit breaker.
• Audit gaps: UPDATE-in-place loses history — compliance asks "what was balance at 14:32 UTC?" and team has no answer.

Why this topic exists

These patterns exist because distributed systems cannot have single-node guarantees:

• No global transaction: 2PC blocks under partition and doesn't scale — sagas provide eventual consistency with explicit compensation.
• Read/write asymmetry: Most systems are 100:1 read:write — CQRS optimizes each path independently.
• Audit and replay: Event sourcing provides immutable history — regulators and debugging both need "what happened?" not "what is?"
• At-least-once delivery: Message brokers guarantee delivery, not exactly-once processing — outbox bridges DB and broker atomically.
• Failure amplification: Without circuit breakers, one slow dependency slows entire call graph — bulkhead and fail-fast contain blast radius.

Core concepts

Pattern definitions — when to apply in Java/Spring:

• Saga: Sequence of local transactions across services; each step has compensating action on failure. Orchestration (central coordinator) vs choreography (events). Example: OrderSaga — reserve inventory → charge payment → ship; compensate refund + release inventory on failure.
• CQRS: Command Query Responsibility Segregation — write model (normalized, transactional) vs read model (denormalized, optimized for queries). Sync via events or polling. Spring: separate @Service for commands vs queries, different repos or even different DBs.
• Event Sourcing: Store events (OrderCreated, PaymentCaptured) not current state; aggregate rebuilt by replay. Enables audit trail, temporal queries, event-driven projections. Spring: event store table + aggregate loader.
• Transactional Outbox: Write business row + outbox row in same DB transaction; separate poller/CDC publishes to Kafka. Guarantees at-least-once without dual-write. Debezium outbox pattern or custom poller.
• Circuit Breaker: Monitor dependency failure rate; OPEN state fails fast without calling dependency; HALF_OPEN probes recovery. Resilience4j @CircuitBreaker in Spring Boot — fallback or cached response.

Internal architecture

Order flow — saga + outbox + CQRS + circuit breaker combined:

┌─────────────── Write Path (Command) ───────────────────────────────┐
│  POST /orders                                                       │
│    → OrderCommandService (@Transactional)                           │
│         ├─ INSERT orders + INSERT outbox_events (same TX)  [Outbox]│
│         └─ emit OrderCreated event                                   │
│                                                                      │
│  Saga Orchestrator (or choreographed via Kafka)              [Saga] │
│    Step 1: InventoryService.reserve()  ──fail──▶ release()        │
│    Step 2: PaymentService.charge()     ──fail──▶ refund()         │
│    Step 3: ShippingService.schedule()                              │
│                                                                      │
│  PaymentService ──@CircuitBreaker──▶ ExternalPaymentGateway [CB]   │
└──────────────────────────────────────────────────────────────────────┘

┌─────────────── Read Path (Query) ──────────────────────────────────┐
│  GET /orders/{id}/summary                                           │
│    → OrderQueryService → order_summary_read_model (denormalized)[CQRS]│
│    populated by OrderProjector listening to OrderCreated events      │
└──────────────────────────────────────────────────────────────────────┘

┌─────────────── Event Store (optional) ───────────────────────────────┐
│  events: [{type:OrderCreated,...}, {type:PaymentCaptured,...}] [ES] │
│  aggregate state = fold(events)                                    │
└──────────────────────────────────────────────────────────────────────┘

Distributed patterns — five diagrams:

Code walkthrough

Spring Boot implementation — outbox, saga step, CQRS query, circuit breaker:

• Outbox: Same @Transactional as business write — crash-safe publish intent.
• Saga compensation: Every forward step returns compensating action — orchestrator runs compensate stack on failure.
• CQRS projector: @EventListener or Kafka consumer updates read model — lag measured and monitored.
• Circuit breaker: OPEN after failure threshold — checkout fails fast with clear error, thread pool preserved.

// ═══════════════════════════════════════════════════════════════
// 1. TRANSACTIONAL OUTBOX — atomic order + event intent
// ═══════════════════════════════════════════════════════════════
@Entity
class OutboxEvent {
    @Id UUID id;
    String aggregateType;
    String aggregateId;
    String eventType;
    String payload; // JSON
    Instant createdAt;
    boolean published;
}

@Service
class OrderCommandService {
    @Transactional
    public OrderId createOrder(CreateOrderCommand cmd) {
        Order order = orderRepo.save(new Order(cmd));
        outboxRepo.save(new OutboxEvent(
            UUID.randomUUID(), "Order", order.getId(),
            "OrderCreated", toJson(cmd), Instant.now(), false));
        return order.getId();
    }
}

// Debezium or @Scheduled poller reads unpublished outbox → Kafka

// ═══════════════════════════════════════════════════════════════
// 2. SAGA — compensating payment step
// ═══════════════════════════════════════════════════════════════
@Service
class PaymentSagaStep {
    private final PaymentClient paymentClient;

    public SagaResult charge(OrderId orderId, Money amount) {
        try {
            paymentClient.charge(orderId, amount);
            return SagaResult.success(new Compensation("refund", orderId));
        } catch (PaymentException e) {
            return SagaResult.failure(e);
        }
    }

    public void compensate(Compensation c) {
        paymentClient.refund(c.orderId()); // idempotent refund
    }
}

// ═══════════════════════════════════════════════════════════════
// 3. CQRS — separate read model projector
// ═══════════════════════════════════════════════════════════════
@Entity
class OrderSummaryView { /* denormalized: id, customer, items, total, status */ }

@Component
class OrderProjector {
    @EventListener
    public void on(OrderCreatedEvent e) {
        summaryRepo.save(buildSummary(e));
    }
}

@Service
class OrderQueryService {
    public OrderSummaryView getSummary(OrderId id) {
        return summaryRepo.findById(id); // fast read, no joins across services
    }
}

// ═══════════════════════════════════════════════════════════════
// 4. CIRCUIT BREAKER — Resilience4j
// ═══════════════════════════════════════════════════════════════
@Service
class PaymentClient {
    @CircuitBreaker(name = "paymentGateway", fallbackMethod = "chargeFallback")
    public void charge(OrderId id, Money amount) {
        restClient.post("/charge", new ChargeRequest(id, amount));
    }

    private void chargeFallback(OrderId id, Money amount, Throwable t) {
        throw new PaymentUnavailableException("Gateway down — fail fast", t);
    }
}

Production example

Debezium outbox — production Kafka integration:

• CDC vs poller: Debezium reads WAL — lower latency than @Scheduled SELECT unpublished.
• Topic routing: Outbox envelope maps to Kafka topics by event type — decouples schema from broker config.
• Consumer idempotency: At-least-once delivery requires dedup — processed event ID store or natural idempotency.

# Outbox table schema (PostgreSQL)
CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

# Debezium connector — captures outbox INSERTs → Kafka topic
# routing: aggregate_type + event_type → topic name
# Example: Order + OrderCreated → order.events topic

# Consumer idempotency — Kafka at-least-once delivery
@KafkaListener(topics = "order.events")
public void consume(ConsumerRecord<String, String> record) {
    if (processedIds.contains(record.key())) return; // dedup
    sagaOrchestrator.onOrderCreated(deserialize(record.value()));
    processedIds.mark(record.key());
}

Enterprise case study

Shopify — outbox at Black Friday scale: Shopify processes peak commerce traffic where every order must reliably trigger inventory, payment, fulfillment, and analytics pipelines. Early architecture used dual-write (DB + Kafka) — intermittent crashes caused "ghost orders" visible in analytics but missing from fulfillment. Shopify adopted the transactional outbox pattern with MySQL binlog CDC to Kafka. During 2023 BFCM, outbox lag stayed under 200ms at 80k orders/minute because publish rate decoupled from request path. Combined with saga-style fulfillment workflows and circuit breakers on third-party payment rails, the platform contained partner outages without stopping checkout globally.

• Problem: Dual-write between MySQL and Kafka — at-least-once on both sides still left gap on process crash.
• Solution: Transactional outbox + CDC; saga compensation for multi-step fulfillment; CB on payment partners.
• Result: Zero lost order events in postmortem window; partner outage isolated to affected payment method.
• Java parallel: Spring @Transactional outbox + Debezium is standard enterprise stack for same problem.

Performance considerations

Pattern performance trade-offs:

• Event sourcing replay: Rebuilding aggregate from 10k events slow — use snapshots every N events.
• CQRS lag: Read model stale 100ms–2s — acceptable for catalog; not for read-your-writes without primary routing.
• Outbox polling: CDC near real-time; poller batch size affects latency vs DB load.
• Circuit breaker overhead: Negligible — ring buffer of call results in memory; saves orders of magnitude vs hung threads.
• Saga orchestration: Central orchestrator is bottleneck at extreme scale — choreography scales better but harder to debug.

Security considerations

Distributed pattern security:

• Outbox payload: Don't store PII in outbox if Kafka topics have broader ACL — encrypt or reference by ID.
• Saga compensation auth: Compensating refund must authenticate same as forward charge — prevent forged compensate calls.
• Event sourcing immutability: Events never deleted — GDPR erasure requires tombstone events and projection redaction.
• Circuit breaker fallback: Fallback must not bypass authorization — cached stale data still scoped to user.

Scalability considerations

Scaling each pattern:

• Saga: Choreography scales horizontally via partitioned Kafka topics; orchestrator scales with sharded saga instances by order_id.
• CQRS: Read model in Elasticsearch/Redis scales independently; write model stays normalized on Postgres.
• Event sourcing: Event store append-only scales with partition by aggregate_id; snapshots prevent replay storms.
• Outbox: Partition outbox table or use multiple pollers with SKIP LOCKED — avoid single-threaded bottleneck.
• Circuit breaker: Per-dependency breaker config — payment gateway separate from fraud service.

Production challenges

Common distributed pattern failures:

• Non-idempotent compensation: Refund called twice on retry — double credit to customer.
• CQRS without monitoring lag: Users see stale order status for minutes — no alert on projector backlog.
• Event sourcing everywhere: Simple CRUD admin panel with ES — team drowns in event versioning complexity.
• Circuit breaker too aggressive: OPEN on first timeout during deploy — flapping; tune failureRateThreshold and waitDurationInOpenState.
• Saga deadlock: Compensating step fails — saga stuck; need manual intervention queue and alerting.

Common mistakes

• Using 2PC/XA across microservices — blocks, doesn't survive partition, avoid in cloud-native Java.
• Event sourcing entire platform when only audit module needs it — YAGNI applies to ES.
• CQRS without defined consistency SLA — "eventually" becomes "never" when projector silently fails.
• Outbox without monitoring unpublished count — backlog grows until events are hours stale.
• Circuit breaker with no fallback strategy — fail fast but user gets opaque 500 instead of degraded UX.

Debugging guide

Debug distributed workflow failures:

• Saga stuck: Trace saga_id across services — find step that didn't compensate; check dead letter queue.
• Outbox backlog: SELECT COUNT(*) FROM outbox WHERE published=false — poller/CDC health, Kafka broker lag.
• CQRS stale read: Compare write model timestamp vs read model updated_at — projector consumer lag metric.
• Circuit breaker flapping: Resilience4j metrics — state transitions OPEN/HALF_OPEN; correlate with dependency deploy.
• Event sourcing mismatch: Replay aggregate in staging — compare rebuilt state vs production snapshot.

# Kafka consumer lag (CQRS projector)
kafka-consumer-groups.sh --bootstrap-server $BROKER \
  --describe --group order-projector

# Resilience4j circuit breaker state
curl localhost:8080/actuator/circuitbreakers

# Outbox backlog query
SELECT COUNT(*), MIN(created_at) FROM outbox_events WHERE published = false;

Best practices

• Use transactional outbox for every DB + message publish — never dual-write.
• Design saga steps idempotent with compensations tested in CI — chaos inject step failure.
• CQRS: define max read lag SLO; alert on projector consumer lag.
• Event sourcing: snapshot aggregates every 100–500 events; version event schema with upcasters.
• Circuit breaker: separate config per dependency; expose state via Actuator; meaningful fallback UX.
• Log correlation ID (saga_id, trace_id) across all services — distributed tracing mandatory.
• Start with orchestrated saga for clarity; move to choreography when scale demands.

Anti-patterns

• Dual-write: save() then kafka.send() outside transaction — guaranteed eventual inconsistency.
• Distributed XA: Atomikos across 5 services — operational nightmare, use saga.
• ES for CRUD: Event sourcing a config table with 3 fields — massive over-engineering.
• Global saga timeout without compensation: Leave inventory reserved forever.
• CB fallback returns success with fake data: User thinks payment succeeded — catastrophic.

Staff engineer notes

• Staff reviews ask: "What happens when step 3 fails after step 1 and 2 succeeded?" — if no compensation answer, design rejected.
• Outbox is table stakes for Java microservices in 2024 — teams still dual-write because "Kafka is reliable" misses the crash window.
• CQRS is not microservices requirement — apply when read/write shape diverges significantly, not by default.
• Event sourcing shines for audit-heavy domains (finance, healthcare) — not for every bounded context.
• Circuit breaker without bulkhead is half a solution — thread pool isolation completes the resilience story.

Interview questions

Hands-on exercise

Lab: Implement mini outbox + saga compensation

• Run playground — observe idempotent saga step with compensation on failure.
• Add OutboxEvent insert in same transaction as order create (in playground simulation).
• Simulate step 2 failure — verify compensation on step 1 runs.
• Add CircuitBreaker wrapper — fail step 2 three times, observe fast-fail on fourth.
• Sketch CQRS read model updated by OrderCreated event.
• Document: what monitoring alerts would you add for outbox backlog and saga stuck count?

Press Run to execute. Real javac + JVM via remote API. Falls back to simulator on network failure.

Architecture trade-offs

• Saga vs 2PC: Saga — available, eventual; 2PC — consistent, blocking, fragile.
• CQRS complexity vs scale: CQRS wins at read/write divergence; loses on simple CRUD domains.
• ES audit vs ops cost: Full replay power vs snapshot/versioning infrastructure.
• Orchestration vs choreography: Orchestration debuggable; choreography scales.

Summary

Distributed patterns let Java microservices achieve correctness without impossible global transactions. You can now design saga flows with compensation, wire transactional outbox with Debezium, split CQRS paths, and protect dependencies with Resilience4j circuit breakers — grounded in Shopify-scale case study lessons.

Key takeaways

• Saga — local TX + compensating actions; no global 2PC.
• CQRS — separate read/write models; monitor projection lag.
• Event sourcing — immutable events as truth; snapshots for replay perf.
• Outbox — atomic DB write + publish intent; CDC to Kafka.
• Circuit breaker — fail fast, prevent cascade; meaningful fallback UX.