Event-Driven Architecture Fundamentals

Advanced System Design & Software Architecture · Lesson 09

Lesson Overview

Checkout at TechLearningPro can authorize a shopper in a few hundred milliseconds of database work plus the acquirer call. The list of things that should happen afterward is longer every quarter: create a shipment, post to finance, award loyalty, send a receipt, update the instructor dashboard, notify a marketplace partner. The first implementation called those collaborators inside PlaceOrderService. A slow warehouse insert delayed the HTTP response. A loyalty outage failed the request after the card was approved. A new consumer required an edit to checkout.

An event is a fact the writer has already committed: OrderAuthorized happened. Consumers react without checkout knowing their names. That is the shift this lesson makes. Commands still exist — “refund this payment” is a command with a sender and a desired outcome. Queries still exist. Events are for facts that fan out. Choreography lets each consumer react on its own. Orchestration puts one process owner in charge when the steps have a required order and a compensation. Kafka is the example transport when the consumer is another process or another team that must not share the checkout transaction. Inside one modular monolith, lesson 7’s @ApplicationModuleListener may be enough. The design of the fact, the key, and the failure policy is the same either way.

This lesson designs the order lifecycle after authorization, previews the transactional outbox so a crash cannot lose the fact, and shows an idempotent consumer. It does not replace Message Queues and Event Streaming, Idempotency and Exactly-Once Processing, or Distributed Transactions and the Saga Pattern. Those go deeper on the broker, delivery claims, and long-running compensations. The enterprise track’s Choreography, Orchestration, Event Brokers, Kafka Architecture, Domain Events, and Outbox Pattern are the parallel reading.

Learning Objectives

By the end of this lesson, you should be able to:

1. Distinguish a command, a query, and an event, and classify OrderAuthorized and RefundPayment correctly.
2. Choose choreography for fan-out after authorization, and orchestration for a refund that must compensate in a defined order.
3. Partition a Kafka topic by order id and explain what ordering Kafka actually guarantees.
4. Write an outbox row in the same database transaction as the order update, and say why @TransactionalEventListener(AFTER_COMMIT) is not that guarantee.
5. Consume with an idempotency table in the same transaction as the consumer’s own write, using Spring Kafka’s @KafkaListener.
6. Configure a finite retry and a dead-letter path with DefaultErrorHandler and DeadLetterPublishingRecoverer.
7. Describe what the shopper sees under eventual consistency, including a UI that does not pretend the shipment exists in the authorization response.
8. Reject a dual write that updates PostgreSQL and calls KafkaTemplate.send without an outbox, and reject a consumer that treats a redelivery as a second shipment.

Prerequisites

• Lesson 8, domain-driven design: OrderAuthorized is already a fact on the aggregate. This lesson delivers it.
• Lesson 7, modular monolith: in-process listeners versus a broker. The business boundary stays the bounded context.
• Lesson 6, ports and adapters: KafkaTemplate lives in a driven adapter. The aggregate does not import it.
• PostgreSQL transactions, and the idea that a commit is the moment a fact becomes durable.
• Spring for Apache Kafka on the classpath (KafkaTemplate, @KafkaListener). You do not need to operate a multi-datacenter cluster for this lesson. You do need the vocabulary: topic, partition, key, offset, consumer group.

Detailed Core Concepts

Commands, events, and queries

Naming is part of the contract. OrderAuthorized is past tense and specific. OrderUpdate is a bucket that forces every consumer to parse a status field and guess. ProcessOrder is a command disguised as a topic. Commands have a target. Events do not have a single target; if you find yourself listing the consumers inside the producer’s method, you are still in the old PlaceOrderService.

Martin Fowler’s split between event notification and event-carried state transfer matters for the payload. A notification says “order 8b2c was authorized; come and ask if you care.” It stays small and it couples consumers to checkout’s query API, which must still answer for old orders. Event-carried state transfer puts the data consumers need on the event: shopper id, line snapshots, total, authorization id. TechLearningPro’s OrderAuthorized is event-carried state transfer for fulfillment and finance, because those contexts must not join the orders schema (lesson 7) and should keep working when checkout is briefly down after the event is in the log. The payload is still not a dump of every column. Payment tokens, PAN, and internal version columns stay off it.

Choreography and orchestration

Choreography means each participant subscribes to facts and decides what it will do. Checkout publishes OrderAuthorized. Fulfillment creates a shipment. Finance posts revenue. Loyalty awards points. Checkout’s code does not mention those modules. Adding a partner webhook is a new consumer group, not a checkout release. The cost is operational: there is no single method you can read to see the whole lifecycle, and a missing consumer is a silent gap until a metric shows it. You need correlation ids, a timeline in tracing, and a written list of consumers next to the event’s schema.

Orchestration means one process owner sends commands and reacts to replies. A refund at TechLearningPro is orchestrated. The steps have a required shape: stop the shipment if it has not left, then refund the payment, then release loyalty, then record the posting. If the shipment has already left, the process takes a different branch (return merchandise) and must not refund before that policy says so. An orchestrator, RefundProcess, stores the step it has reached. It sends CancelShipment and waits for ShipmentCancelled or ShipmentAlreadyShipped. It sends RefundPayment and waits for PaymentRefunded or a decline. Timeouts belong to the orchestrator, because someone must decide what “stuck” means. Consumers do not secretly start the next step.

Use choreography when the reactions are independent and the business can tolerate them completing at different times. Use orchestration when there is a business process with compensations, timeouts, and a person who owns the stuck case. A system can use both: choreography after authorization, orchestration for refund. Choosing one style for every interaction is how refunds become a tangle of events nobody can compensate, or how a simple receipt send grows a process manager.

The saga pattern in Distributed Transactions and the Saga Pattern is the long-running version of this orchestrated refund. This lesson stops at the process state and the command/event split.

What Kafka is doing in this design

Kafka is a durable, partitioned log. Producers append records to a topic. A topic has partitions. Records with the same key go to the same partition, and a partition is an ordered log. A consumer group cooperates so each partition is read by one member of the group at a time. Fulfillment uses group id fulfillment. Finance uses finance. Each group has its own position (offset). Fulfillment being behind does not block finance.

Ordering guarantee you can rely on: for one key, in one partition, records are in producer order. Across partitions there is no global order. Key OrderAuthorized by orderId. Then fulfillment sees that order’s events in order. You do not get “all events in the system in the order they happened,” and you should not design a consumer that needs it.

Delivery: with the usual consumer configuration, processing is at least once. A consumer can see the same record twice if it crashes after handling and before the offset is committed. Producer idempotence (enable.idempotence=true, with acks=all) stops the producer from appending duplicates when it retries inside a single producer session. It does not make the consumer exactly-once. End-to-end safety in this lesson is the outbox plus a consumer deduplication key. Kafka transactions (transactional.id) are a real, sharper tool for consume-transform-produce pipelines; they are not required to learn the lifecycle, and they are not a substitute for a business idempotency key if two different events can both create a shipment.

Spring’s APIs for this lesson:

• org.springframework.kafka.core.KafkaTemplate to send.
• @KafkaListener(topics = "orders.order-authorized", groupId = "fulfillment") to consume.
• org.apache.kafka.clients.consumer.ConsumerRecord when you need the key, topic, and offset.
• org.springframework.kafka.listener.DefaultErrorHandler and DeadLetterPublishingRecoverer for retries and dead letters.
• org.springframework.util.backoff.FixedBackOff.

Those types are the transport adapter. The event class OrderAuthorized stays a Java record in the checkout API with no Kafka imports.

The dual-write problem and the outbox

This sequence loses facts:

textCopy
commit order as AUTHORIZED
KafkaTemplate.send(...)

If the process dies after the commit and before send, the order is authorized and fulfillment never hears it. If you send before the commit and the commit fails, fulfillment creates a shipment for an order that rolled back. @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT) runs the send after a successful commit in the same process. It closes the “send then roll back” window. It does not close the crash window: the listener is not durable. A kill -9 between commit and listener drops the event. That annotation is appropriate for a cache refresh you can rebuild. It is not appropriate for OrderAuthorized.

The transactional outbox puts the message in the same database transaction as the order update:

textCopy
BEGIN
  update orders.checkout_order set status = 'AUTHORIZED' ...
  insert into orders.outbox (id, aggregate_id, event_type, payload, created_at)
  COMMIT

A separate publisher reads rows where published_at is null, sends them, and sets published_at after the broker has accepted the record. A crash retries the send. Consumers must tolerate duplicates, because the publisher can send and then crash before updating published_at. That is at least once, and it is honest.

Debezium (or a similar log-based CDC tool) can publish from the database log instead of a polling publisher. That is an alternative implementation of the same pattern, with its own operational cost. This lesson’s walkthrough uses a poller and KafkaTemplate so the control flow stays visible in your code. The enterprise chapter Outbox Pattern compares the options. Do not mark a row published before the send succeeds. Do not delete the row as your only acknowledgement; a published_at timestamp leaves you an audit trail.

Eventual consistency the UI can tell the truth about

After HTTP 201, the order is AUTHORIZED in checkout. The shipment row might not exist yet. The confirmation page says the payment was authorized and fulfillment is being prepared. It does not say “shipment S-19 created” unless that response actually waited for fulfillment, which this design deliberately does not. Support tools show the outbox row and the consumer offset lag so a missing shipment is diagnosable. Eventual consistency is a specified delay, not a shrug. State the target: shipment row within 30 seconds at p95 under normal load, alert when the oldest unpublished outbox row is older than two minutes, alert when a consumer group’s lag exceeds a threshold you set with fulfillment.

Idempotent consumers

The business key is the event id (a UUID stored in the outbox and in the payload), not the Kafka offset. Offsets are not stable identifiers across reprocessing from a new group. Fulfillment inserts into fulfillment.consumed_event (consumer_name, event_id) in the same transaction as the shipment insert. A unique violation means the event was already applied; the consumer commits the Kafka offset and moves on. Deduplicating in memory, or in a cache with a TTL, forgets the key on restart and double-ships.

Architecture and Design Diagrams

mermaidCopy
flowchart LR
  subgraph checkout [Checkout context]
    O[Order aggregate]
    OB[(orders.outbox)]
    O --> OB
  end
  PUB[Outbox publisher]
  OB --> PUB
  PUB --> T[(topic orders.order-authorized)]
  T --> F[Fulfillment group]
  T --> N[Finance group]
  T --> L[Loyalty group]
  F --> FS[(fulfillment.shipment)]

mermaidCopy
sequenceDiagram
  participant U as PlaceOrderService
  participant DB as PostgreSQL orders
  participant P as Outbox publisher
  participant K as Kafka
  participant F as Fulfillment listener
  U->>DB: UPDATE order AUTHORIZED
  U->>DB: INSERT outbox
  Note over U,DB: one commit
  P->>DB: SELECT unpublished
  P->>K: send key=orderId
  K-->>P: ack
  P->>DB: SET published_at
  K->>F: record
  F->>F: insert consumed_event and shipment

mermaidCopy
flowchart TB
  subgraph choreography [After authorization]
    E[OrderAuthorized] --> F1[Fulfillment]
    E --> F2[Finance]
    E --> F3[Loyalty]
  end
  subgraph orchestration [Refund]
    R[RefundProcess] --> C1[CancelShipment command]
    C1 --> R
    R --> C2[RefundPayment command]
    C2 --> R
    R --> C3[ReverseLoyalty command]
  end

Trust boundary: the event carries shopper id and line snapshots. It does not carry the payment-method token. The Kafka principal the publisher uses is a service account that can write orders.order-authorized and cannot read unrelated topics. Consumers use their own accounts with read on that topic only. Payload retention follows the privacy review for shopper data; a log is still a store.

Technical Explanation

From aggregate fact to outbox row

Order.markAuthorized appends OrderAuthorized in memory, as in lesson 8. The application service, inside the transaction that updates the order to AUTHORIZED, asks a port:

javaCopy
public interface Outbox {
    void append(UUID eventId, String aggregateType, UUID aggregateId, String eventType, String payloadJson);
}

The JDBC adapter inserts into orders.outbox. The event id is generated before the call and stored on the event so a retry of the use case that re-reads an already authorized order does not append a second business fact. The application service only appends when it transitions the status. A replayed HTTP request returns the stored result and does not append again.

Payload JSON is produced by a small serializer in the adapter or in the application package. Keep the schema explicit: eventId, orderId, shopperId, lines[{sku, quantity, unitAmount, currency}], total, authorizationId, occurredAt. Add a schemaVersion integer. Consumers ignore unknown fields and reject an unsupported version to the dead-letter path rather than guessing.

Publisher

A scheduled task in the checkout module, every second, loads a batch of unpublished rows ordered by created_at, sends each with KafkaTemplate.send("orders.order-authorized", orderId, payload).get(2, TimeUnit.SECONDS), then sets published_at. get waits for the producer ack so you do not mark a row published on a buffered failure. A send failure leaves published_at null and the next poll retries. Use a FOR UPDATE SKIP LOCKED claim if you run more than one publisher instance, so two instances do not double-send quite so eagerly. Even with that lock, at-least-once remains the contract, because a crash after ack and before published_at is still possible. Say that in the runbook. Do not “fix” it by committing the offset and the business write in two uncoordinated steps on the consumer.

Producer configuration worth setting explicitly: acks=all, enable.idempotence=true. Those are Apache Kafka producer configs, set in Spring Boot via spring.kafka.producer.acks and spring.kafka.producer.properties.enable.idempotence. They make producer retries safe inside the client. They do not replace the outbox.

Consumer

javaCopy
@Component
class ShipmentOnAuthorized {
    private final FulfillmentJdbc jdbc;

    ShipmentOnAuthorized(FulfillmentJdbc jdbc) { this.jdbc = jdbc; }

    @KafkaListener(topics = "orders.order-authorized", groupId = "fulfillment")
    @Transactional
    void onAuthorized(ConsumerRecord<String, String> record) {
        OrderAuthorized event = json.read(record.value());
        jdbc.applyOnce(event);
    }
}

applyOnce inserts the consumed-event row and the shipment. @Transactional on the listener, with a KafkaTransactionManager or with Spring Kafka’s container transaction configuration, is the setting that ties the database commit to the offset commit. The Spring for Apache Kafka reference describes container transactions and AckMode. If you are not ready to enable Kafka transactions, still do the database deduplication, and accept that a crash after the database commit and before the offset commit redelivers, hits the unique key, and returns. That combination is the one this lesson recommends as the baseline. It uses features you can test with Testcontainers and PostgreSQL.

Retries and dead letters

A poison payload that always throws should not block the partition forever. Spring Kafka’s DefaultErrorHandler backs off and then recovers. The sample in the Spring Kafka repository wires it as a CommonErrorHandler bean, which Boot attaches to the listener container:

javaCopy
@Bean
CommonErrorHandler errorHandler(KafkaOperations<Object, Object> template) {
    return new DefaultErrorHandler(
            new DeadLetterPublishingRecoverer(template),
            new FixedBackOff(1_000L, 2L));
}

FixedBackOff(interval, maxAttempts) with 2L retries the delivery twice after the first failure (three attempts), one second apart, matching the Spring Kafka reference’s description of that constructor. DeadLetterPublishingRecoverer then publishes the record to the dead-letter topic (by default, the original topic name plus .DLT). A human or a tool inspects orders.order-authorized.DLT. Infinite retries on a bad payload stall every later event for that partition, which means every later event for other orders that hashed to the same partition. That is a checkout incident caused by one bad message. Finite retries plus a dead letter are the production default. Classify exceptions: a database outage can use a longer back-off; a JsonParseException should not retry ten times.

Refund orchestration, sketched

RefundProcess is an aggregate in the checkout or a dedicated returns context. States: STARTED, SHIPMENT_CANCEL_REQUESTED, PAYMENT_REFUND_REQUESTED, COMPLETED, WAITING_FOR_RETURN. It stores the order id and the last command id. The driving adapter is a support or shopper action, a command StartRefund. The orchestrator sends CancelShipment on a command topic fulfillment.commands keyed by order id, or calls fulfillment’s HTTP API if fulfillment is still in-process. Replies come back as events. The orchestrator is the only component that decides the next command. Fulfillment does not refund the card because it saw a cancel. Payment does not cancel the shipment because it saw a refund. That single-writer of the process is the point of orchestration.

Timeouts: if ShipmentCancelled has not arrived in ten minutes, the process moves to NEEDS_ATTENTION and pages the returns squad. It does not invent a refund. Automated compensation without an owner is how money leaves the building.

What stays synchronous

Price lookup, seat reservation, and payment authorization stay synchronous for the shopper, as lessons 6 and 7 designed. Events start after the fact the shopper already waited for. Publishing PriceRequested and waiting for PriceQuoted on a broker adds latency and failure modes to a call that is an in-process method. Event-driven architecture is a choice per interaction, not a coating on the whole system.

Practical Implementation or Design Walkthrough

Problem. After an order is authorized, fulfillment, finance, and loyalty must react without checkout calling them. A crash must not lose the fact. A redelivery must not create two shipments. Refunds of an authorized order must follow a defined sequence.

Assumptions. PostgreSQL holds orders.outbox. Kafka is the transport between processes. One topic orders.order-authorized, six partitions to start (enough for 40 orders per second with headroom; revisit with real lag, not with a slogan). JSON payloads, schemaVersion 1. Spring Boot, spring-kafka, KafkaTemplate<String, String>.

1. Outbox table

sqlCopy
create table orders.outbox (
    id uuid primary key,
    aggregate_type text not null,
    aggregate_id uuid not null,
    event_type text not null,
    payload jsonb not null,
    created_at timestamptz not null,
    published_at timestamptz
);
create index outbox_unpublished on orders.outbox (created_at) where published_at is null;

2. Append inside the authorizing transaction

The same TransactionTemplate that calls orders.markAuthorized calls outbox.append. Both use the thread-bound connection. A test rolls back the template and asserts the outbox row is absent. A second test commits and asserts the row is present even when a fake publisher is down. That test is the design. If it is hard to write, the send is still inside the use case via KafkaTemplate directly.

3. Publish

javaCopy
public void publishBatch() {
    for (OutboxRow row : store.lockUnpublished(100)) {
        SendResult<String, String> result = kafkaTemplate
                .send("orders.order-authorized", row.aggregateId().toString(), row.payload())
                .get(2, TimeUnit.SECONDS);
        store.markPublished(row.id(), result.getRecordMetadata().offset());
    }
}

KafkaTemplate.send(topic, key, value) returns a CompletableFuture (Spring Kafka 3). get with a timeout surfaces a failure to the scheduler, which will try the row again. Store the offset on the row if you want a support query; the source of truth for “was it handled” is still the consumer’s dedup table, not this offset.

4. Fulfillment dedup

sqlCopy
create table fulfillment.consumed_event (
    consumer_name text not null,
    event_id uuid not null,
    primary key (consumer_name, event_id)
);
create table fulfillment.shipment (
    order_id uuid primary key,
    created_at timestamptz not null
);

applyOnce inserts both. The shipment’s primary key on order_id is a second guard if two different event ids ever claimed the same order. Investigate that case; do not silently create two shipments with a random id.

5. Error handler

Register the CommonErrorHandler bean shown above. Add a metric for dead-letter publishes. A test can point a listener at an embedded or Testcontainers broker, send a payload with schemaVersion: 99, and expect a record on orders.order-authorized.DLT after the back-off. Do not point the dead-letter topic back at the same consumer group without a human step.

6. Refund process state

A table returns.refund_process (order_id primary key, state, updated_at). Transitions are methods on a RefundProcess aggregate: start, onShipmentCancelled, onShipmentAlreadyShipped, onPaymentRefunded. The application service sends the next command only after save. This is orchestration because the next command is chosen by this aggregate, not by a web of listeners.

Expected result. Killing the publisher after commit leaves rows with published_at null, and a restart drains them. Replaying one record creates one shipment. The shopper’s HTTP handler does not inject fulfillment’s repository. A refund in PAYMENT_REFUND_REQUESTED does not send a second RefundPayment if that method was already recorded; the command uses the order id as its idempotency key.

Enterprise Real-World Scenarios

Scenario 1: Authorization fan-out at 40 orders per second

Peak place-order traffic is 40 per second, each producing one OrderAuthorized. Three consumer groups keep up easily at that rate; the risk is not throughput. The risk is a finance consumer that writes to a slow ledger and falls behind, growing lag, while fulfillment stays current. Because groups are independent, shoppers still get shipments. Finance lag is an alert to the finance squad, not a checkout outage. The topic key is the order id so a corrective OrderCancelled later lands on the same partition after the authorization record. Consumers apply events for one order in offset order and use the dedup table so a replayed authorization during a rebalance does not post revenue twice. The confirmation page reads checkout’s own database, not the Kafka log, and the copy was written with product: “Payment authorized. We’re preparing your items.” Loyalty points may appear a minute later. Product signed that sentence. Engineering does not “fix” the copy by blocking the HTTP response on all three consumers.

Scenario 2: A refund that must not race itself

A shopper requests a refund for an authorized book order. Sometimes the warehouse has not picked it. Sometimes the carrier has it. Choreographed listeners — payment refunds because it saw RefundRequested, fulfillment cancels because it saw the same event — refunded two orders that were already on a truck and failed to cancel the pick on a third because fulfillment’s consumer was down while payment’s was not.

The orchestrated process replaced that. RefundProcess.start records STARTED and sends CancelShipment only. Payment is not informed yet. Fulfillment replies ShipmentCancelled or ShipmentAlreadyShipped. On cancel, the process sends RefundPayment with the original authorization id and the same order id as the idempotency key. On already shipped, the process moves to WAITING_FOR_RETURN and does not refund until a return arrives. Each transition is one transaction in returns. A double-click on the refund button hits the primary key on order_id and returns the current state. The shopper sees “refund started,” not a synchronous card reversal, unless the process completes within the HTTP wait, which you should not promise. Support sees the state column. That visibility is the operational reason to orchestrate: a stuck refund has an owner and a state, not three topics to correlate by hand at midnight.

Scenario 3: The dual write that passed a demo

A team under deadline called kafkaTemplate.send immediately after orderRepository.save, in the same method, without a transaction, and mocked Kafka in tests. The demo worked. The first production deploy overlapped with a broker blip. Several hundred orders reached AUTHORIZED with no outbox and no event. Fulfillment’s backlog looked healthy because it had nothing to consume. The repair was a backfill job that read AUTHORIZED orders with no matching shipment and republished from the order row — possible only because the order row itself held the snapshot the event would have carried. The lasting fix was the outbox insert in the same transaction, and a test that fails the build if KafkaTemplate appears in the application service. The backfill remains as a runbook for one release. It is not the architecture.

Best Practices

• Name events in the past tense, with a noun the glossary already has. Version the payload.
• Put on the event the snapshots consumers must store. Leave tokens, secrets, and unused columns off it.
• Key the Kafka record by the aggregate id that defines order.
• Treat delivery as at least once. Deduplicate with a unique business key in the consumer’s database, in the same transaction as the side effect.
• Use an outbox in the writer’s transaction for facts other contexts must not miss.
• Keep @TransactionalEventListener(AFTER_COMMIT) for rebuildable side effects, and document which is which.
• Give each consumer group its own lag alert and its own dead-letter topic monitoring.
• Bound retries. Send poison messages to a dead-letter topic with DeadLetterPublishingRecoverer.
• Use choreography for independent reactions. Use an orchestrator with explicit state when the next step depends on the previous reply.
• Keep shopper-critical reads and authorizations synchronous. Do not make the confirmation page a Kafka consumer.
• Correlate logs with eventId and orderId. Those two fields should be on every consumer log line.
• Plan a backfill from the system of record. The order row, not the topic, is the authority if the log is wrong or young.

Common Mistakes and Anti-Patterns

Dual write. Database commit and KafkaTemplate.send as two best-effort steps. One succeeds. Use the outbox.

Event as a command to a named service. A topic fulfillment-do-shipment that checkout publishes, with fulfillment as the only legal consumer, is a command channel. That can be right for orchestration. Calling it an event and then expecting unknown future consumers to understand an imperative payload confuses both styles. Name commands as commands and publish them to the handler’s channel. Publish events as facts.

Global order assumptions. A consumer that merges finance and fulfillment by “whichever message arrives first” will mis-order work across partitions. Order per key only. If you need a total order of all retail activity, you do not have it; build a time-windowed report from the databases.

In-memory dedup. A HashSet of event ids in the listener process empties on deploy and then creates duplicate shipments during the redelivery that deploys cause. The unique key has to be durable.

Infinite retry on the hot partition. A single bad JSON stalls every order whose id hashes to that partition. Finite FixedBackOff and a dead letter. Wake someone up for dead-letter depth, not for a tight retry loop.

Choreographed refund. Independent listeners that each “do the right thing” on RefundRequested will do it in race order. Put a RefundProcess in charge.

Giant payload “so nobody needs to call us.” A 400 KB event with nested product HTML and a card token is a privacy incident and a consumer-coupling incident. Send the snapshot the consumers persist, and let them query their own models afterward.

Broker inside one process as theater. If every consumer is a bean in the same deployable and the fact need not survive a separate failure domain, lesson 7’s module listener is the simpler outbox-like mechanism (Modulith’s publication registry). Introduce Kafka when a consumer is another runtime, another team’s release train, or a replayable log you have already decided to operate.

Troubleshooting and Design Review

Design review for a new event:

• Is this a fact or a command?
• Who must not miss it, and is the outbox in the same transaction as that fact?
• What is the Kafka key, and which ordering story does it support?
• What does the consumer write, and where is the idempotency key?
• What happens on poison payload, and who watches the dead letter?
• What does the shopper see before the consumer has run?
• Which fields are forbidden on the payload?

Interview Questions and Answers

Q1. What is the difference between a command and an event?

A command asks a specific handler to change state and may be rejected: RefundPayment can be declined. An event states something that has already been committed: OrderAuthorized is not a request fulfillment may veto inside checkout’s transaction. Checkout publishes it because the order aggregate and the database already accepted the authorization. Commands have an owner. Events have consumers that are unknown to the publisher’s business logic. If the publisher lists every consumer in code, the interaction is still a procedure with extra steps. Past-tense names help. ProcessRefund on a “topic of events” is usually a command that will be mistaken for a fact.

Q2. When do you choose choreography, and when orchestration?

Choreography fits independent reactions to a fact. After authorization, fulfillment, finance, and loyalty do not need to run in a fixed order and do not compensate each other. Each subscribes, does its write, and deduplicates. Orchestration fits a process with branches, compensations, and timeouts. A refund must ask fulfillment before it asks payment, and must do something different if the parcel has shipped. A RefundProcess aggregate stores the step and is the only sender of the next command. Choreography’s failure mode is a silent missing consumer. Orchestration’s failure mode is a god process that absorbs every new step. Use both in one system, on purpose, and write down which lifecycle is which.

Q3. What ordering guarantees does Kafka give you here?

Records with the same key append to the same partition, and a partition is ordered. Key by orderId, and fulfillment sees OrderAuthorized before a later OrderCancelled for that order, assuming a single producer pipeline that preserves that send order. There is no total order across orders. Consumer groups do not share offsets: finance can be behind fulfillment. Rebalancing can redeliver the last records, which is why “Kafka order” does not remove the need for idempotency. If a design needs a global sequence of every retail event, Kafka partitions will not provide it.

Q4. Why is a transactional outbox required if you already have AFTER_COMMIT?

@TransactionalEventListener(phase = AFTER_COMMIT) runs only if the transaction committed, in that process. It prevents publishing a fact that rolled back. If the JVM stops after commit and before the listener sends, the fact is in PostgreSQL and not in Kafka. Nothing will retry, because the listener’s memory is gone. The outbox inserts the payload in the same transaction as the order update. A publisher retries until the broker acks, then sets published_at. You still have at-least-once delivery, so consumers dedupe. You no longer have “committed order, zero chance to publish.” Use AFTER_COMMIT for a derived cache. Use the outbox for OrderAuthorized.

Q5. How do you implement the outbox with Spring and PostgreSQL without inventing a framework?

A table orders.outbox with a UUID primary key, aggregate id, event type, JSON payload, created_at, and nullable published_at. The use case inserts a row with JdbcClient inside the same TransactionTemplate as the status update. A scheduler selects unpublished rows, calls KafkaTemplate.send(topic, key, payload).get(...), and updates published_at only after that future completes. More than one publisher instance should claim rows with FOR UPDATE SKIP LOCKED. Debezium is a valid alternative that tails the log; it is not required to explain or to ship the first version. There is no Spring annotation that means “outbox” in Spring Framework itself. Spring Modulith’s event publication registry solves a related problem for in-process listeners. For Kafka, the table is the design.

Q6. How does the consumer avoid creating two shipments?

The listener and the database share a transaction. The handler inserts (consumer_name, event_id) into fulfillment.consumed_event and inserts the shipment keyed by order_id. Either unique constraint stops a duplicate. A redelivery after a crash hits the constraint and returns successfully so the offset can move. Dedup keys live in PostgreSQL, not in a heap Set. The event id is the id written to the outbox, carried in the JSON. The Kafka offset is a transport position, not a business key. Idempotency and Exactly-Once Processing goes further; this is the baseline that matches at-least-once delivery.

Q7. What does enable.idempotence on the producer actually buy?

It lets the Kafka producer retry without appending the same record twice within its session, when used with the required ack settings (acks=all and a nonzero in-flight window that the client constrains). It protects the log from producer-side duplicates. It does not deduplicate an outbox publisher that sends, receives an ack, crashes before published_at, and sends again: those are two logical sends from the client’s point of view if the producer id changes, and the consumer must still dedupe. It also does not deduplicate two different events that both mean “create a shipment.” Say the sentence in review so nobody ticks a box labeled exactly-once and deletes the consumed_event table.

Q8. How should retries and poison messages work in Spring Kafka?

A record that throws should be retried a small number of times, then removed from the hot path. DefaultErrorHandler with new FixedBackOff(1_000L, 2L) waits one second and retries twice after the original attempt. DeadLetterPublishingRecoverer publishes the failed record to a dead-letter topic, by default suffixed .DLT, using a KafkaOperations instance. Register the handler as a CommonErrorHandler bean. Do not loop forever: one bad payload blocks its partition, and that partition carries other orders. Alert on dead-letter depth. A parse error and a database outage do not deserve the same back-off; classify them if the volume justifies it.

Q9. What should the shopper see if fulfillment is ten seconds behind?

The authorization response comes from checkout’s database: payment authorized, order id, no tracking number. The page copy states that preparation follows. A later read can show shipment data once fulfillment has committed it. Blocking the POST until finance, loyalty, and fulfillment have all consumed couples their outages to the sale and throws away the reason you published an event. Eventual consistency is acceptable here because the shopper’s money decision is already durable in checkout and payment. It would not be acceptable for the authorization itself, which remains a synchronous port call. The product sentence and the alert on consumer lag are part of the design, not follow-up polish.

Q10. How do you test this without flattering mocks?

Test one: roll back a transaction and assert no outbox row; commit and assert a row while the publisher bean is disabled. Test two: run the publisher against a Kafka Testcontainers cluster and assert a record with the order id as key. Test three: invoke the consumer handler twice with the same event id and assert one shipment. Test four: a schema version the consumer rejects lands on the .DLT topic. Mocking KafkaTemplate inside a test that never inserts the outbox will recreate scenario 3. The broker in a testcontainer is slower than a mock and it is the right tool for the publisher contract. Domain tests of RefundProcess transitions need no broker at all.

Q11. When would you keep an in-process event instead of Kafka?

When every consumer lives in the same deployable, shares the same release, and can use Spring Modulith’s @ApplicationModuleListener with the publication registry (lesson 7). You still design a fact, a key, and an idempotent handler. You do not operate a cluster. Move to Kafka when a consumer must deploy independently, when a team outside the JVM must integrate, or when you need a retained log for replay. Moving early “for practice” means checkout’s availability includes the broker’s availability. That can be the right trade once those consumers exist. It is an expensive rehearsal when they do not.

Hands-On Exercises

Exercise 1: Order lifecycle with domain events

Problem. Design the lifecycle of a TechLearningPro retail order using domain events after authorization, plus a refund path.

Tasks.

1. List facts versus commands. Include OrderAuthorized and the refund steps.
2. Draw choreography for authorization fan-out and orchestration for refund. Name the owner of refund state.
3. Specify the Kafka topic, key, and a versioned payload. Mark forbidden fields.
4. Define the outbox table and the moment of insert relative to the order update.
5. Define the fulfillment consumer’s dedup and shipment keys.
6. Specify retries, dead letter, and the lag alert.
7. Write the shopper-visible copy and the p95 target for shipment creation.
8. State what enable.idempotence does not cover.

Expected deliverables. Sequence sketch, payload example, table definitions, failure policy, UI sentence.

Acceptance. A dual write cannot satisfy the design. A second delivery of the same event id cannot create a second shipment. Refund does not start at payment.

Exercise 2: Review a dual-write pull request

The change commits the order and then calls kafkaTemplate.send in a try that logs and swallows. Write the review: the failure window, the replacement outbox flow, and the one test you require before merge. Keep the same topic name.

Rubric.

Lesson Summary

Events are committed facts. Commands are requests a specific context may refuse. After TechLearningPro authorizes an order, fulfillment, finance, and loyalty react by choreography, each at its own pace, keyed by order id on a Kafka topic. Refunds are orchestrated because the next step depends on the previous reply and someone must own the timeout. The fact reaches Kafka through an outbox row committed with the order, not through a best-effort send and not through an after-commit listener alone. Consumers deduplicate in the same database transaction as their own write. Retries are finite and poison messages land on a dead-letter topic. The shopper’s authorization response stays synchronous and does not pretend the shipment already exists. Producer idempotence makes broker retries safer and still leaves the business key in place. Later lessons go into the broker, sagas, and exactly-once claims. You now have a lifecycle you can operate.

Further Reading

• Spring for Apache Kafka, Sending messages and Annotation-based error handling — KafkaTemplate, DefaultErrorHandler, DeadLetterPublishingRecoverer, FixedBackOff.
• Apache Kafka, Producer idempotence and the consumer groups section of the documentation.
• Spring Framework, @TransactionalEventListener — what AFTER_COMMIT covers.
• Martin Fowler, What do you mean by “Event-Driven”? — notification versus event-carried state transfer.
• Microsoft, Transactional outbox pattern — the dual-write problem and the table.
• Debezium documentation, Outbox event router — the CDC alternative named in this lesson.
• Enterprise track: Choreography, Orchestration, Outbox Pattern, Kafka Architecture, Event Brokers, Domain Events.
• Sibling lessons: Domain-Driven Design Fundamentals, Layered and Modular Monolith Architecture, Message Queues and Event Streaming, Distributed Transactions and the Saga Pattern, Idempotency and Exactly-Once Processing.