Message Queues and Event Streaming

Advanced System Design & Software Architecture · Lesson 33 · High-Level Design and Distributed Systems

Lesson Overview

TechLearningPro’s checkout commits OrderAuthorized and an outbox row in one PostgreSQL transaction. Event-Driven Architecture Fundamentals stopped at that fact: a publisher calls KafkaTemplate.send, fulfillment deduplicates, and a poison payload must leave the partition. This lesson is about the transport underneath that fact, and about the transports that should not be Kafka.

A launch morning produces two very different workloads. The first is a log of facts. Finance, fulfillment, loyalty, and a new analytics job all need OrderAuthorized, each at its own pace, and analytics must be able to start today and read the last ninety days. The second is a unit of work. “Send this enrollment email through the provider” should be handled by one worker, deleted when the provider accepts it, and parked when the address is invalid. Replaying yesterday’s emails because a new consumer subscribed would be a product incident.

Those are a log and a queue. Apache Kafka is the log in this design: an append-only, partitioned, retained record stream. A queue, in the sense RabbitMQ implements with a queue bound to an exchange, holds a message until a consumer acknowledges it, then removes it. Spring for Apache Kafka (KafkaTemplate, @KafkaListener, DefaultErrorHandler, DeadLetterPublishingRecoverer) and Spring AMQP (RabbitTemplate, @RabbitListener) are the Java adapters. The decision is which delivery, ordering, and retention story the business operation actually has.

The default you should design for on both transports is at least once. A crash after the side effect and before the acknowledgement redelivers the record. Exactly-once is a separate design in Idempotency and Exactly-Once Processing. This lesson covers the retry budget, the dead-letter path, consumer-group scaling, and lag. Those pipelines become the integration backbone of Microservices Architecture Principles and Enterprise Integration Patterns.

Learning Objectives

By the end of this lesson, you should be able to:

1. Choose a retained log or a competing-consumer queue from the retention, replay, and ownership requirements, and name what each choice costs.
2. Explain Kafka partitions, keys, offsets, and consumer groups, including why a group cannot run more active consumers than it has partitions.
3. State the delivery semantic you actually get with Spring Kafka’s default acknowledgement: at least once, with duplicates after a crash or a rebalance.
4. Configure a finite retry and a dead-letter topic with DefaultErrorHandler and DeadLetterPublishingRecoverer, and separate retryable provider failures from poison payloads.
5. Set a prefetch or a max.poll.records bound so a slow handler cannot pull an unbounded amount of work it will not finish before max.poll.interval.ms.
6. Read consumer lag as log-end offset minus committed offset, and decide whether to scale consumers, add partitions, or slow the producer.
7. Keep ordering where the key provides it, and keep a business idempotency key where the transport will redeliver.
8. Hand the same pipeline to a second team without a shared database, which is the bridge into the microservices module.

Prerequisites

• Lesson 9, Event-Driven Architecture Fundamentals: outbox, @KafkaListener, and the limit of producer enable.idempotence. This lesson starts at the broker.
• Lesson 32, Distributed Transactions and the Saga Pattern: a saga step travels on this transport and must tolerate redelivery.
• Lesson 31, CAP Theorem and Consistency Models: partition order is not a global total order.
• Lesson 26, Load Balancing and Traffic Management: a consumer group assigns whole partitions, which is stricter than per-request HTTP balancing.
• PostgreSQL unique constraints, and the Spring types in spring-kafka and spring-amqp. The vocabulary you need is topic, partition, offset, consumer group, acknowledgement, and dead letter.

Detailed Core Concepts

A queue removes work; a log retains facts

A queue stores a message so that one consumer can take it. RabbitMQ is the reference here because the vocabulary is stable and Spring AMQP is the usual Java adapter. A producer publishes to an exchange. A binding routes to a queue. A consumer receives the message. An acknowledgement deletes it. A second consumer on the same queue competes for the next message. That is competing consumers, and it matches “send this enrollment email once.”

A log appends and keeps records until retention deletes them. Kafka’s topic is split into partitions. Each partition is an ordered sequence of records with increasing offsets. Reading does not delete. Independent consumer groups keep independent positions. Fulfillment can sit at the tip while analytics, which subscribed this morning, reads from the start of a ninety-day retention.ms with cleanup.policy=delete. Compaction (cleanup.policy=compact) keeps the latest record per key, which fits a status snapshot and destroys an audit of every authorization.

Replay is the operational test. On the log you start a group or reset offsets and read what retention still holds. On the queue the acknowledgement already deleted the message. A second copy had to be bound before publish. Neither broker is a workflow engine. The application supplies the retry budget, the idempotency key, and the poison-message policy.

What a Kafka record, a key, and a partition actually decide

A Kafka record has a topic, a partition, an offset, a key, a value, a timestamp, and optional headers. The producer chooses the partition. With a non-null key, the default partitioner hashes the key (Murmur2 in the Apache Kafka client) and maps it to a partition: partition = hash(key) % numberOfPartitions. Records with the same key therefore append to the same partition, and a partition is ordered. Key OrderAuthorized and OrderCancelled by orderId and fulfillment sees that order’s records in producer order, provided one producer pipeline preserves send order for that key (max.in.flight.requests.per.connection kept within the limit idempotence requires, which is at most 5, and retries enabled).

A null key does not give you “round robin forever.” Modern Apache Kafka producers use a sticky partitioner for null keys: they fill a batch to one partition, then switch. That improves batching and throughput. It does not give per-entity order. If the business rule is “this learner’s emails stay in order,” the key is the learner id. If there is no order requirement, a null key is legitimate and often faster.

Ordering stops at the partition boundary. There is no broker operation that means “deliver every topic in the order the business events happened.” A consumer that joins two partitions and sorts by timestamp is doing its own merge, and timestamps can tie or skew. Design the key for the order you can name in one sentence. “All of learner 42’s enrollment emails in send order” is a key. “Every email in the company in send order” is not a Kafka guarantee.

Partitions are also the unit of parallelism and the unit of failure isolation. One slow or poison record delays its partition. Other partitions continue. That is why a dead-letter path is not a nicety. A single corrupt payload on a 24-partition topic blocks one twenty-fourth of the key space for as long as you retry it in place.

Adding partitions later changes the hash mapping. Keys that used to share a partition can split. If you rely on order, plan the partition count for the peak consumer concurrency you intend, and treat a partition increase as a migration with a dual-read or a new topic, not as a casual config bump during the launch. Kafka does not reshuffle old records onto the new partitions. Only new appends use the new modulus.

Consumer groups, offsets, and the idle consumer

A consumer group is a set of consumers that share one group.id and cooperate so that each partition is assigned to at most one member. Fulfillment uses fulfillment. Finance uses finance. They do not share offsets. Each group’s position is stored in the internal __consumer_offsets topic. The committed offset is the next offset the group should read. If the handler processed offset 5, the commit records 6. Quoting “we committed 5” without that convention causes off-by-one incidents in replay tools.

If a group has more members than partitions, the extra members stay idle. Six consumers and four partitions means two processes do no work. Scaling the deployment without increasing partitions does not increase parallelism past the partition count. The reverse is useful: twelve partitions and three consumers means each consumer owns about four partitions. You can add consumers later, up to twelve, without a topic migration.

Assignment is recomputed when membership changes. That rebalance happens on startup, shutdown, a crash, or a missed heartbeat. Heartbeats use heartbeat.interval.ms and session.timeout.ms. A separate timer, max.poll.interval.ms (default 300000), is the maximum time between poll calls. A handler that processes a batch for longer than that interval is assumed dead, the group rebalances, and the same records are delivered to someone else. The handler may still be running. You now have two workers in the side effect unless the side effect is idempotent. This is the practical reason to bound batch size (max.poll.records, default 500) and to keep a single record’s work well under the poll interval.

Two assignment protocols matter in operations. The eager protocol (classic RangeAssignor) revokes all partitions, stops the world, and assigns again. The cooperative sticky protocol (org.apache.kafka.clients.consumer.CooperativeStickyAssignor) revokes only the partitions that must move. Set the strategy explicitly. Client defaults have changed across Kafka releases, and a launch is a bad time to discover which one the classpath picked. Cooperative sticky rebalances shorten the window where a partition has no owner. They do not remove duplicates: a partition that moves can redeliver the last uncommitted records.

Spring Kafka’s listener container commits offsets according to ContainerProperties.AckMode. BATCH commits after the poll batch is processed. RECORD commits after each record. MANUAL and MANUAL_IMMEDIATE require the listener method to accept an Acknowledgment and call acknowledge(). The container turns off the client’s enable.auto.commit and commits itself. Auto-commit on a timer, which is the raw consumer default (enable.auto.commit=true), can commit offsets for records the handler has not finished, or can fail to commit records it has finished. Leave acknowledgement to the container, and make the handler finish the side effect before it returns.

auto.offset.reset applies when a group has no committed offset. latest skips history. earliest reads from the beginning of retention. none throws if no offset exists, which is the right setting for a financial consumer that must not silently start at the tip. Analytics, on its first day, wants earliest and a dedicated group id so it does not move fulfillment’s offsets.

Delivery semantics you can claim

Producer idempotence (enable.idempotence=true, which has been the producer default since Apache Kafka 3.0, together with acks=all) stops the producer from appending the same batch twice when it retries within a producer session. The broker remembers the producer id and sequence. It does not stop an outbox relay from sending the same row twice after a crash, because that second send is a new sequence from the client’s point of view if the first one was acknowledged and then lost in the relay’s own bookkeeping. Consumers still dedupe.

acks=all waits for the full in-sync replica set. Pair it with a topic min.insync.replicas of at least 2 on a replication factor of 3, or the “all” acknowledgement can mean “the leader alone” when only one replica is in sync. That configuration is a durability choice from lesson 29’s replication discussion applied to the log.

Retries, poison messages, and the dead letter

A transient failure and a poison message need different policies. The email provider returns 503 or 429: the same payload may succeed in a few seconds. The payload fails JSON parsing, or the provider returns 400 because the address is malformed: retrying it on the same partition blocks every other learner whose key landed there.

Spring Kafka’s DefaultErrorHandler seeks back to the failed record so earlier offsets in the batch can be committed, then applies a BackOff. new FixedBackOff(2_000L, 3L) waits two seconds and allows three retry attempts after the original delivery. ExponentialBackOff exists in org.springframework.util.backoff when the provider’s recovery is slower. addNotRetryableExceptions takes exception types that skip the back-off and go straight to the recoverer. Put serialization and conversion failures there. Put a dedicated InvalidRecipientException there. Leave provider timeouts on the retry path.

DeadLetterPublishingRecoverer publishes the failed ConsumerRecord with a KafkaOperations (a KafkaTemplate). The default destination topic is the original name plus .DLT. Headers such as KafkaHeaders.DLT_ORIGINAL_TOPIC, DLT_ORIGINAL_PARTITION, DLT_ORIGINAL_OFFSET, and DLT_EXCEPTION_MESSAGE record why it moved. The main consumer’s offset can then advance. The partition is free.

The in-container back-off delays that partition for the sleep interval. A five-minute sleep inside the listener is the wrong tool: use a short back-off, then the dead letter, and let an operator or a rate-limited replay consumer republish. @RetryableTopic is the Spring Kafka feature that publishes to delay topics instead of sleeping the live partition. It is appropriate when delays must be long and the main partition must keep moving. It is more moving parts. Start with DefaultErrorHandler plus a dead letter unless the back-off you need would stall a launch partition.

RabbitMQ’s equivalent is a dead-letter exchange. The queue is declared with the arguments x-dead-letter-exchange and, when you need it, x-dead-letter-routing-key. The consumer rejects the message without requeue. Spring AMQP’s AmqpRejectAndDontRequeueException does that. A retry queue can dead-letter back to the work queue after a TTL, which is the broker-side delay. Cap the retry count with a header the handler increments, or the message loops forever. The dead-letter queue is then an operator inbox, with an alert on depth, the same way the Kafka .DLT topic is.

Backpressure: the broker will not save you from a slow handler

Kafka producers append as fast as the partition leader accepts them, until disk, quotas, or max.block.ms stop the client. There is no per-topic “queue is full, reject the publisher” switch comparable to a RabbitMQ queue with x-max-length and overflow=reject-publish. Consumer lag is your backpressure signal. Lag for a partition is the log end offset minus the group’s committed offset. Spring and Micrometer expose this through the Kafka client metrics (records-lag-max and related consumer metrics). Alert on lag age as well as lag count: a million records that are ten seconds of a large partition is different from a million records that represent a six-hour outage.

When lag climbs, make the handler faster, add consumers up to the partition count, add partitions only with a key plan, or slow the producer. Slowing OrderAuthorized grows the outbox, which is visible and durable. A bounded RabbitMQ queue with x-max-length and overflow=reject-publish pushes back into the publisher instead. An unbounded queue hides overload until memory or disk dies.

On the RabbitMQ consumer, prefetch (basic.qos, exposed as SimpleRabbitListenerContainerFactory.setPrefetchCount) caps unacknowledged deliveries. A prefetch of 1 is fair and slow. A prefetch of thousands with an 80 ms provider call means a crash redelivers a huge backlog and a slow consumer hoards work. Size prefetch to what the handler finishes inside its timeout. On Kafka, max.poll.records is the matching knob, together with concurrency on ConcurrentKafkaListenerContainerFactory. Broker produce and fetch quotas, managed through Kafka’s AdminClient, stop a partner producer from filling the disk the checkout log depends on.

When the queue and the log both exist

TechLearningPro keeps both shapes. orders.order-authorized is keyed by orderId, retained for ninety days, and read by fulfillment, finance, loyalty, and later analytics. notifications.email-requested is keyed by recipientId, retained for a few days, and has one sending group, notifications-email. A second group that calls the provider sends a second email. A RabbitMQ queue notifications.email is the alternative when you want broker-side length limits and you have no replay consumer. The outbox makes the request durable with the enrollment commit. The handler’s idempotency row makes the redelivery safe. The walkthrough specifies the Kafka version of the email path.

Architecture and Design Diagrams

mermaidCopy
flowchart LR
  subgraph logPath [Retained log]
    OP[Outbox publisher] --> OT[(orders.order-authorized)]
    OT --> GF[group fulfillment]
    OT --> GN[group finance]
    OT --> GA[group analytics reads history]
  end
  subgraph queuePath [Competing consumers]
    NP[Notification publisher] --> NT[(notifications.email-requested)]
    NT --> GE[group notifications-email]
    GE --> SMTP[Email provider]
    GE --> DLT[(notifications.email-requested.DLT)]
  end

mermaidCopy
sequenceDiagram
  participant C as Consumer thread
  participant K as Partition leader
  participant H as Email handler
  participant DB as sent_notification
  participant P as Email provider
  C->>K: poll
  K-->>C: record offset 481
  C->>H: handle
  H->>DB: insert event_id
  H->>P: send
  P-->>H: 250 accepted
  H-->>C: return
  C->>K: commit offset 482
  Note over C,K: crash before commit redelivers 481
  Note over H,DB: unique event_id makes the second send a no-op

mermaidCopy
flowchart TB
  R[Record fails in listener] --> Q{Retryable?}
  Q -->|Provider 429 or 503| B[FixedBackOff then redelivery]
  B --> R
  Q -->|Parse error or invalid recipient| D[DeadLetterPublishingRecoverer]
  D --> T[(topic.DLT)]
  D --> OK[Offset advances]
  B -->|Attempts exhausted| D

Trust boundary: the email record carries recipient id, template id, and a hashed idempotency key. It does not carry a password reset token in clear text if a narrower channel exists, and it never carries a card number. The Kafka ACL grants the notification publisher WRITE on notifications.email-requested and the email group READ on that topic plus WRITE on the .DLT topic. Analytics has READ on orders.order-authorized and no access to the notification topic. TLS or SASL protects the client connection. The provider call uses the provider’s documented HTTPS API from configuration. This lesson does not invent a mail-vendor SDK.

Technical Explanation

Commit timing and key affinity

The sequence diagram is the delivery semantic. The listener handles offset 481, the handler inserts event_id, the provider accepts, and only then does the container commit 482. A crash before that commit redelivers 481. The unique key makes the second pass a no-op. Acknowledging before the provider call is at most once and drops mail. A rebalance redelivers uncommitted records the same way. Cooperative assignment moves fewer partitions. It does not wrap the provider in a transaction.

Key by recipientId, not by templateId. A launch template hashed onto one partition leaves the other twenty-three idle. Key orders by orderId, not by status. Partition count follows the consumer threads you will actually run, which the walkthrough derives from handler latency.

Priority, delay, and the saga handoff

A Kafka partition has no priority over its neighbors. There is no broker scheduler that always drains partition 0 first. Enrollment mail and marketing mail belong on different topics, or on different RabbitMQ queues, so a marketing backlog cannot occupy the enrollment consumers. A delay belongs on a retry topic, a queue TTL, or a run_at column. Sleeping inside the listener risks max.poll.interval.ms.

Lesson 32’s saga sends a command and waits. The handler’s unique key is the saga id plus the step name, so redelivery does not run the step twice. The log does not know the saga exists. Circuit Breaker, Retry and Bulkhead Patterns then wraps the provider call so a dead dependency cannot hold every consumer thread.

Topics are a security boundary. Kafka ACLs grant producers WRITE and DESCRIBE, consumers READ, and nobody who ships application code ALTER on a shared topic. A partner reads a projected topic, not the raw checkout log. Dead-letter retention is long enough to debug and short enough for the privacy review, and access to .DLT is narrower than access to the main topic because those payloads get pasted into tickets. TLS or SASL is the baseline on the client connection.

Practical Implementation or Design Walkthrough

Problem

During a cohort launch, TechLearningPro must send one enrollment email per learner per course offering. Peak is 200 email requests per second for fifteen minutes. The request is durable once the enrollment transaction commits. Invalid addresses must not block other learners. A provider 503 must be retried a few times. A crash must not drop the request and must not send unbounded duplicates. Analytics does not consume this topic. Order facts remain on their own retained log from lesson 9.

Requirements and constraints

• Functional: publish EmailRequested after enrollment commit; deliver to the provider; record success; surface permanent failures to an operator queue.
• Latency: p95 from outbox insert to provider accept under 30 seconds at the stated peak, excluding provider incidents.
• Ordering: one recipient’s messages stay ordered. No global order.
• Delivery: at least once, with a database idempotency key.
• Scale: 200 requests/second sustained for the launch window; 12 million requests on a peak day is the daily integral the capacity model must also survive, but the partition plan is set by the 200/second spike.
• Constraint: use Apache Kafka client semantics and Spring Kafka types that exist in the library. The provider is called over HTTPS with Spring’s RestClient, base URL from configuration. No fictional mail SDK.

Capacity that sets the partition count

A synchronous provider call at about 80 ms occupies a thread. One thread completes about 12 requests per second. ConcurrentKafkaListenerContainerFactory concurrency of 4 yields about 48 requests per second per process if the CPU and the provider allow it. Five processes yield about 240, which covers 200 with headroom. Each concurrency slot is a consumer in the group, so the group has about 20 members. Partition count must be at least that. Choose 24 partitions, not 200. Twenty-four leaves a little room to add consumers during an incident. Two hundred partitions would add controller metadata, slower rebalances, and more open files on every client for a parallelism level the provider latency does not require.

Retention: 3 days (retention.ms=259200000) so a weekend backlog can drain and a dead letter can be inspected. cleanup.policy=delete. Replication factor 3, min.insync.replicas=2.

The record and the sent table

The value is JSON with eventId, recipientId, offeringId, templateId, and requestedAt. The Kafka key is recipientId. The idempotency key is eventId, a UUID the outbox row already has.

sqlCopy
create table notifications.sent_email (
  event_id uuid primary key,
  recipient_id uuid not null,
  provider_message_id text,
  sent_at timestamptz not null
);

The handler inserts event_id in the same database transaction that records success. A unique violation means this event was already accepted. The handler returns normally so the offset commits. A provider call that succeeded and a process that died before the insert is the remaining window: the provider may have the message, and a retry may send it again. Close that window by sending the provider an idempotency key if their documented API accepts one (many do, as an Idempotency-Key header or a client reference field — use the field their docs name). If their API has no such field, store IN_PROGRESS before the call and reconcile ambiguous timeouts with their status endpoint by eventId. Do not invent the endpoint. Read the provider’s documentation and adapt only what it lists.

Spring Kafka listener and error handler

javaCopy
@Component
class EnrollmentEmailListener {
  private final EmailHandler handler;

  EnrollmentEmailListener(EmailHandler handler) {
    this.handler = handler;
  }

  @KafkaListener(
      topics = "notifications.email-requested",
      groupId = "notifications-email")
  public void onRequested(ConsumerRecord<String, String> record) {
    handler.handle(record.key(), record.value());
  }
}

javaCopy
@Bean
public CommonErrorHandler emailErrorHandler(KafkaTemplate<String, String> template) {
  DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);
  DefaultErrorHandler handler =
      new DefaultErrorHandler(recoverer, new FixedBackOff(2_000L, 3L));
  handler.addNotRetryableExceptions(
      SerializationException.class,
      MessageConversionException.class,
      InvalidRecipientException.class);
  return handler;
}

InvalidRecipientException is ours, thrown on provider 400. A 503 or a RestClient timeout stays retryable. Register the handler with setCommonErrorHandler on this listener factory so checkout’s order consumer does not inherit the email retry budget.

Set max.poll.records=50 so one poll is a few seconds at 80 ms. Keep HTTP timeouts well under max.poll.interval.ms. Set partition.assignment.strategy to org.apache.kafka.clients.consumer.CooperativeStickyAssignor. Use auto.offset.reset=earliest for this documented group id so a new group drains the backlog. Leave commits to the container (enable.auto.commit=false). The outbox relay uses acks=all, enable.idempotence=true, and marks the row published only after KafkaTemplate.send(...).get(...) returns. That acknowledgement means the broker accepted the record. It does not mean the email was sent.

Failure handling and tests

Tests: two handler calls with one event_id produce one sent_email row; a 400 path on Testcontainers Kafka lands on notifications.email-requested.DLT; a rolled-back enrollment writes no outbox row.

orders.order-authorized stays the long-retention, multi-group log keyed by orderId. The notification topic is a short-retention work log with one sending group. If the team later prefers RabbitMQ, publish with RabbitTemplate.convertAndSend to a direct exchange, set prefetch on SimpleRabbitListenerContainerFactory, and reject permanent failures with AmqpRejectAndDontRequeueException toward a queue that sets x-dead-letter-exchange. The sent_email table stays either way. Analytics replay stays on the order log.

Enterprise Real-World Scenarios

Scenario 1: Launch-day enrollment mail at TechLearningPro

Business context. A new cohort opens at 09:00 local time. About 150,000 learners have a saved seat. Enrollment rows commit over roughly fifteen minutes as the gate opens, with a spike of 200 email requests per second. The learner-facing promise is a confirmation email. The finance and fulfillment pipelines from lesson 9 must not slow down because the mail provider is slow, and the mail pipeline must not re-send when analytics replays orders.

Boundaries. Checkout owns the enrollment transaction and the outbox. The notification context owns sent_email, the listener, and the provider credential. The provider is outside the trust boundary. Kafka is the durable buffer between them.

Choices. Separate topics, 24 partitions, key recipientId, one consumer group, DefaultErrorHandler with a three-retry back-off, non-retryable invalid recipients, dead-letter topic, lag alert at two minutes of age, and an idempotency key on the provider call. Marketing mail is a different topic with a smaller consumer allocation so it cannot occupy the enrollment consumers. During the launch, an operator watches group lag and dead-letter depth, not CPU alone. A provider 503 for ten minutes grows lag; the outbox and the log hold the work. A provider 400 for one bad row dead-letters that row.

What would fail the review. One shared “events” topic, auto-commit, infinite retry, and a consumer that sends mail as a side effect of the analytics group.

Scenario 2: A new analytics group on the order log

Business context. Ninety days after launch, the data team builds a revenue dashboard. They need every OrderAuthorized still inside retention. Fulfillment is at the tip of the log. Nobody can replay by asking fulfillment to “resend,” because fulfillment acknowledges by committing offsets and does not store the event for others.

Choices. A new group id analytics-revenue with auto.offset.reset=earliest reads from the start of retention without moving fulfillment. The consumer writes a warehouse load that is idempotent on eventId. It has its own lag alert, because a six-hour analytics delay is not a learner-facing incident, while a six-hour fulfillment delay is. Broker disk for ninety days of orders was purchased when the topic was created, using the capacity method from Capacity Estimation and Workload Modeling: average record size times events per day times retention days times replication factor, plus headroom.

A queue would have needed a second binding on day one or a PostgreSQL export. The log makes the late consumer possible inside retention, at the cost of disk and an ACL that keeps the analytics principal off the notification topic. Fulfillment and analytics will deploy on different schedules without a shared schema. The topic is that contract, and API Design, Versioning and Contracts is where HTTP evolution gets the same treatment: a version field, compatible additions, and a dead letter for an unknown required version.

Best Practices

• Write down the sentence that chooses the transport: “several consumers, replay within retention” is a log; “one worker, delete on success, bounded length” is a queue.
• Key the record by the entity whose order you promise. Keep the partition count near the maximum useful consumer count.
• Commit after the side effect. Store a unique business key before you consider the work done. Assume redelivery.
• Retry a small, classified set of failures. Dead-letter the rest. Alert on dead-letter depth and on lag age.
• Set max.poll.records and HTTP timeouts so processing cannot exceed max.poll.interval.ms. Set RabbitMQ prefetch to a number the handler can finish.
• Give each workload its own topic, group id, error handler, and ACL. Do not multiplex mail and payments on one topic to save a name.
• Set acks=all and min.insync.replicas together. Turn on producer idempotence and still keep the consumer dedupe table.
• Separate enrollment traffic from bulk marketing with different topics or queues so a backlog of one cannot starve the other.
• Treat partition increases and group-id changes as migrations. auto.offset.reset=latest on a new group id skips the backlog.
• Retention, ACLs, and dead-letter access are privacy controls. A log is a datastore. Propagate eventId for support; Spring Kafka’s Micrometer observation support carries the trace.

Common Mistakes and Anti-Patterns

Using a queue because the team already has RabbitMQ, then discovering replay. The analytics team asks for last quarter’s orders. Acknowledgements deleted them. The repair is a PostgreSQL export or a new log, plus a written retention requirement next time. The presence of a broker on the architecture diagram is not a retention decision.

Using Kafka for every call, including request/response that needs a reply in a few hundred milliseconds. A command that needs an answer now is an HTTP or RPC call. Putting it on a topic adds latency, a consumer fleet, and a timeout you must invent. The email request is asynchronous because the enrollment response does not include the provider’s message id. Authorization of the card is synchronous because the shopper is waiting on the decision. Lesson 9 made that split; the transport has to respect it.

Infinite retries on the live partition. The handler catches all exceptions and loops, or the error handler has a back-off with effectively unlimited attempts. One bad record freezes its partition. Every other recipient on that partition waits. Cap attempts, classify exceptions, publish to .DLT, and page a person.

Auto-commit plus a slow side effect. The raw consumer commits on a timer. The offset moves past a record whose provider call has not returned. A crash drops that email. Let the Spring container commit after the listener returns, and do the work inside the listener.

Scaling consumers past partitions and calling it horizontal scale. The extra pods are idle. The group describes them as members without assignment. Either the partition count was the limit you chose, or the key design is so skewed that adding partitions would not help until the key changes.

A shared dead-letter topic with no owner. Every service publishes failures to dead-letter. Nobody alerts. Payloads from payments sit next to marketing. Give each source topic its own dead-letter topic, a named owner, a retention period, and a replay procedure that preserves the idempotency key.

Compacting a fact topic to save disk. Compaction keeps the latest record per key, so an earlier OrderAuthorized disappears when a later status shares the key. Use cleanup.policy=compact for snapshots. Use delete for facts you must replay in full. Long sleeps inside the listener are the same family of mistake: max.poll.interval.ms expires, the partition moves, and a second consumer runs the record. Put delays on a retry topic or a queue TTL.

Troubleshooting and Design Review

Design review for a new topic:

• Who consumes, and will a late consumer need replay? That answer picks log or queue.
• What is the key, in one sentence of ordering?
• What is the partition count, derived from handler latency and peak rate?
• When does the offset or ack happen relative to the side effect?
• What is the idempotency key, and where is the unique constraint?
• Which exceptions retry, which dead-letter, and who owns the dead letter?
• What lag age pages a human, and is that age a learner-facing promise or a batch promise?
• Which fields are forbidden on the payload, and who can read the topic?

Interview Questions and Answers

Q1. When do you pick a queue, and when do you pick a log?

Pick a queue when one worker should perform a side effect and the broker can delete the message on acknowledgement. Competing consumers, a bounded length, and a dead-letter exchange match that job. TechLearningPro’s enrollment email is the example, if no second business consumer needs a copy. Pick a log when several independent groups must read the same fact, or a future group must replay within retention. OrderAuthorized is the example: fulfillment, finance, loyalty, and a later analytics group. The cost of the log is disk, partition planning, and access control. The cost of the queue is that acknowledgement is deletion. State the retention and the consumer list before you name the product.

Q2. What does a Kafka consumer group guarantee?

Members that share a group.id divide the topic’s partitions so each partition is assigned to at most one member at a time. That gives you parallelism up to the partition count and keeps a single partition’s records in order for that group. A second group has its own offsets and reads the same records independently. Extra members beyond the partition count stay idle. The group does not guarantee exactly-once side effects, global order, or that a rebalance will avoid redelivery. The committed offset is the next offset to read.

Q3. Why is at-least-once the default you should design for?

The safe commit order is side effect, then offset commit. A crash between those steps redelivers the record. A rebalance can do the same for uncommitted offsets. Spring Kafka’s container commits after the listener returns when you leave acknowledgement to it, which is this default. Designing for at most once means committing before the work, and a crash drops the work. Exactly-once delivery of an email or a database write is not what the broker acknowledgement means. Make the handler idempotent on a business key, and say “at least once” in the design review.

Q4. What does producer enable.idempotence change for consumers?

It lets the producer retry without appending a duplicate batch inside its session, with acks=all and an in-flight limit the client enforces (at most five). The broker deduplicates by producer id and sequence. An outbox relay that sends, receives the acknowledgement, crashes before it records published_at, and sends again is still the consumer’s problem if that second send is a new append. A second logical event that also creates a shipment is the consumer’s problem. Keep the unique constraint on the consumer. Producer idempotence is necessary for sane producer retries and insufficient for business exactly-once.

Q5. How do DefaultErrorHandler and DeadLetterPublishingRecoverer fit together?

The error handler seeks so a failed record can be retried without losing the ability to commit earlier offsets in the batch. FixedBackOff defines the wait and the number of retry attempts. Exceptions registered with addNotRetryableExceptions skip retries. When attempts are exhausted, or the exception is not retryable, DeadLetterPublishingRecoverer publishes the record using a KafkaTemplate. The default topic name is the source topic plus .DLT, and headers such as KafkaHeaders.DLT_ORIGINAL_OFFSET preserve the source position. The main offset can then advance. A long sleep in the listener is a bad substitute: it risks max.poll.interval.ms and stalls the partition.

Q6. A team doubles the pod count and throughput stays flat. What do you ask?

How many partitions, and how many group members have an assignment? Members beyond the partition count are idle. If one partition holds all the lag, the key is hot (templateId instead of recipientId). If every member is busy and latency is the provider call, more threads blocked on HTTP will not help until the call is faster or the provider allows more concurrency. Add pods only after assignment and handler time say the extra consumers can do work.

Q7. How do you keep a poison message from blocking a partition?

Classify it. Parsing failures and permanent business rejections are not retryable. Transient provider failures retry a small number of times. After that, publish to the dead-letter topic and advance. Alert on dead-letter depth. Give an operator a replay that sends the same eventId so the idempotency key still holds. An unbounded retry, a catch that swallows and loops, or a “skip on any error” that commits without a record of the skip are the two failures: stuck partitions, or silent data loss. The dead letter is the middle path.

Q8. What is consumer lag, and when is it an incident?

Lag is the difference between the latest offset on the partition and the group’s committed offset. Ten thousand records at 200 per second is about a minute of enrollment mail, which may be inside the promise. The same count on fulfillment at a quiet hour can be a stalled consumer. Alert on age and on your product promise, not on a single magic number copied from another system. Lag that grows while consumers are busy is a capacity problem. Lag that grows while consumers are idle is a poison message, a rebalance loop, or a handler blocked on I/O.

Q9. Why is max.poll.interval.ms a correctness setting?

If the listener does not poll within that interval, the member is considered dead and its partitions move. The old member may still be inside a provider call. The new member receives the same records. You get duplicate side effects and, under load, a rebalance storm. Bound max.poll.records, set HTTP client timeouts below the interval, and do not sleep for long delays inside the handler. The interval is not a retry policy. It is a liveness timeout with delivery consequences.

Q10. How does this design change when checkout and notifications become separate services?

The outbox stays in the enrollment database so the request is committed with the enrollment. The notification service owns the group, sent_email, and the provider credential, and it does not query the enrollment schema. The topic name, key, and schema are the contract, versioned like an HTTP API. ACLs replace in-process trust. Notification failure shows up as lag and dead letters, not as a failed enrollment response. That is the split Microservices Architecture Principles and Enterprise Integration Patterns build on. Lesson 41’s circuit breaker then protects the consumer threads from a dead provider.

Hands-On Exercises

Exercise 1: Design the enrollment mail pipeline

Problem. Specify the transport for TechLearningPro enrollment email at 200 requests per second for fifteen minutes, with invalid addresses and provider 503s, without disturbing the retained order log.

Tasks.

1. State whether this workload is a log, a queue, or a log used as a work queue. Name the topic or queue, the key, and the retention.
2. Derive a partition count or a competing-consumer count from an 80 ms provider call and the peak rate. Show the arithmetic.
3. Define the payload fields and the fields that must not appear.
4. Specify offset or ack timing, the sent_email unique key, and the provider idempotency field you will look up in their docs.
5. Specify retryable versus non-retryable failures, the dead-letter destination, and the alert.
6. Explain what happens if a second consumer group subscribes and calls the provider.
7. State how analytics should read OrderAuthorized without sending email.

Expected deliverables. A short decision record, the diagram of publisher, topic, group, provider, and dead letter, the table definition, and the failure table.

Acceptance. A crash after a successful provider response cannot create unbounded sends. A poison payload cannot stall the whole topic. A new analytics consumer cannot move the fulfillment offsets or send mail.

Exercise 2: Review a “we will just use Kafka for everything” proposal

The proposal puts card authorization, enrollment email, and the analytics feed on one topic with a null key, enable.auto.commit=true, and a listener that retries all exceptions forever. Write the review. Identify what is synchronous, what must be retained, what the key should be, how duplicates happen, and which Spring Kafka types replace the infinite retry. Keep the critique specific to this lesson’s mechanisms.

Rubric.

Lesson Summary

A queue deletes work when a competing consumer acknowledges it. A log retains records so each consumer group can move at its own pace and a later group can replay within retention. Kafka partitions are the unit of order, parallelism, and failure isolation; the key selects the partition, and the group cannot run more busy consumers than it has partitions. The commit-after-handling default is at least once, so the handler needs a unique business key. Producer idempotence makes producer retries safe and still leaves consumer duplicates in place. Finite, classified retries and a dead-letter topic keep a poison record from stalling a partition. Lag age, max.poll.records, max.poll.interval.ms, and RabbitMQ prefetch are how you stop a slow handler from taking on work it cannot finish. TechLearningPro keeps the order fact on a long-retention topic with several groups, and keeps enrollment email on a separate workload with one sending group. When those consumers become separate services, the topic is the contract: the microservices module adds independent deployability, versioned schemas, and client-side resilience around the same log.

Further Reading

• Apache Kafka documentation, Topics, partitions, and offsets and the consumer-group and producer-config sections, including enable.idempotence, acks, and max.poll.interval.ms.
• Spring for Apache Kafka, Annotation-driven listener endpoints and Annotation-based error handling — DefaultErrorHandler, DeadLetterPublishingRecoverer, FixedBackOff, CommonErrorHandler.
• RabbitMQ documentation, Consumer acknowledgements and prefetch and Dead letter exchanges — basic.qos, x-dead-letter-exchange.
• Spring AMQP, Annotated endpoint — @RabbitListener, AmqpRejectAndDontRequeueException.
• Martin Kleppmann, Designing Data-Intensive Applications, the log and stream-processing chapters — retained logs versus traditional messaging.
• Enterprise track: Event Brokers, Kafka Architecture, Dead Letter Channel, Competing Consumers.
• Next in this path: API Gateway and Backend for Frontend, Idempotency and Exactly-Once Processing, then Module 4 Microservices Architecture Principles, Circuit Breaker, Retry and Bulkhead Patterns, and Enterprise Integration Patterns.