Kafka

Introduction

Apache Kafka is the event streaming platform behind LinkedIn, Uber, Netflix, and most Java microservices architectures. Producers publish records to topics partitioned for parallelism; consumers in consumer groups read with at-least-once or exactly-once semantics.

Java engineers interact with Kafka via spring-kafka, the official kafka-clients library, or Kafka Streams. This lesson covers production fundamentals: partition strategy, consumer group rebalancing, offset management, and exactly-once processing with idempotent producers and transactional APIs.

Business problem

Why synchronous microservices fail at scale:

• Cascade failures: Order service synchronously calls inventory, payment, notification — one slow service blocks all.
• Peak load: Black Friday order spike — HTTP thread pools exhaust; events buffer in Kafka instead.
• Audit and replay: Regulators require event log — REST calls leave no durable history.
• Multiple consumers: Same OrderCreated event feeds inventory, analytics, fraud — REST requires fan-out from producer.
• Decoupled deploy: New fraud service subscribes to existing topic — zero changes to order producer.

Why this topic exists

Kafka solves the "n×m integration problem" — n producers × m consumers without point-to-point wiring:

• Log abstraction: Durable append-only log — producers write; consumers read at their pace with offset bookmarks.
• Partitions: Parallelism — topic split into ordered partitions; scale consumers up to partition count.
• Consumer groups: Each partition consumed by one consumer in group — horizontal scale with automatic rebalancing.
• Retention: Events replayable for days — new consumer rebuilds state from history.
• Exactly-once: Idempotent producer + transactions — payment events processed once despite retries.

Core concepts

Kafka core concepts for Java developers:

• Producer: Sends records to topic partition — key determines partition (hash(key) % numPartitions).
• Consumer: Polls batches from assigned partitions; commits offset after processing.
• Partition: Ordered sequence within partition — no global order across partitions.
• Consumer Group: Group ID — Kafka assigns each partition to one consumer; add consumer → rebalance.
• Offset: Position in partition log — auto-commit vs manual commit for at-least-once control.
• Exactly-once: enable.idempotence=true + transactional producer + read_committed consumer isolation.

Internal architecture

Kafka cluster topology for Java services:

Topic: order.events (6 partitions, replication-factor 3)

  Producer (Order Service)                Consumer Group: inventory-workers
       │ key=orderId                           ├─ Consumer 1 → P0, P1
       ▼                                       ├─ Consumer 2 → P2, P3
  ┌─────────┐    ┌─────────┐                   └─ Consumer 3 → P4, P5
  │ Broker 1│    │ Broker 2│    ┌─────────┐
  │ P0 L    │    │ P1 L    │    │ Broker 3│    Consumer Group: analytics
  │ P1 F    │    │ P2 L    │    │ P0 F    │    └─ Consumer 1 → all partitions
  └─────────┘    │ P3 L    │    │ P2 F    │       (separate group = independent read)
                 └─────────┘    └─────────┘

Exactly-once flow:
  producer.initTransactions()
  producer.beginTransaction()
  producer.send(orderEvent)
  producer.sendOffsetsToTransaction(offsets, consumerGroupMetadata)
  producer.commitTransaction()

Producer → partition → consumer group flow:

Code walkthrough

Spring Kafka producer and consumer — order events with manual ack:

• ENABLE_IDEMPOTENCE: Broker deduplicates retries via producer ID and sequence numbers.
• Key = orderId: All events for one order land in same partition — ordered processing.
• Manual ack: Offset committed only after DB write succeeds — at-least-once without data loss on crash.
• groupId: Multiple inventory service instances share load — one partition per consumer max.

@Configuration
public class KafkaConfig {
    @Bean
    public ProducerFactory<String, OrderEvent> producerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);  // exactly-once producer
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        return new DefaultKafkaProducerFactory<>(props);
    }
}

@Service
public class OrderEventPublisher {
    private final KafkaTemplate<String, OrderEvent> kafka;

    public void publish(OrderEvent event) {
        // key = orderId ensures all events for order go to same partition
        kafka.send("order.events", event.orderId(), event);
    }
}

@Component
public class InventoryConsumer {
    @KafkaListener(topics = "order.events", groupId = "inventory-workers",
                   containerFactory = "manualAckFactory")
    public void onOrderCreated(OrderEvent event, Acknowledgment ack) {
        reserveStock(event.orderId(), event.items());
        ack.acknowledge();  // commit offset after successful processing
    }
}

Production example

Exactly-once consume-process-produce with KafkaTransactions:

• transaction-id-prefix: Enables transactional producer — begin/commit around listener.
• read_committed: Consumers skip messages from aborted transactions.
• EOS listener: Spring Kafka ties consumer offset to outgoing publish in one transaction.
• Trade-off: Higher latency than at-least-once — use for payment/fraud, not analytics.

@Bean
public ConcurrentKafkaListenerContainerFactory<String, OrderEvent> eosFactory(
        ConsumerFactory<String, OrderEvent> cf,
        KafkaTemplate<String, ProcessedEvent> template) {
    var factory = new ConcurrentKafkaListenerContainerFactory<String, OrderEvent>();
    factory.setConsumerFactory(cf);
    factory.getContainerProperties().setAckMode(AckMode.RECORD);
    factory.setKafkaTemplate(template);  // enables EOS listener
    return factory;
}

@KafkaListener(topics = "order.events", groupId = "fraud-check",
               containerFactory = "eosFactory")
public void process(OrderEvent event) {
    FraudScore score = fraudEngine.score(event);
    // Transactional: offset commit + output publish atomic
    kafkaTemplate.send("fraud.results", event.orderId(), new FraudResult(score));
}

// application.yml
spring.kafka:
  producer:
    transaction-id-prefix: fraud-tx-
  consumer:
    isolation-level: read_committed  // skip aborted transactions

Enterprise case study

LinkedIn — Kafka origin: LinkedIn built Kafka to handle real-time activity feeds — billions of events/day. Java services produce page views, job applications, messaging events. Key lesson: partition count is a capacity planning decision — increasing partitions later is painful; plan for peak consumer throughput upfront.

• Before: Point-to-point queues — adding consumer required producer changes.
• After: Kafka log — unlimited consumers at their own pace; 7-day retention for replay.
• Java stack: Kafka Streams for aggregations; spring-kafka for service integration.
• Ops: Monitor consumer lag — alert when lag > 10K messages for payment topics.

Performance considerations

Kafka performance tuning for Java:

• Batching: linger.ms=5, batch.size=32KB — amortize network round-trips.
• Compression: lz4 or zstd on producer — CPU for bandwidth trade-off worth it.
• Partition count: Start with #partitions ≥ peak consumer instances; order key avoids hot partitions.
• Consumer poll: max.poll.records and processing time < max.poll.interval.ms — avoid rebalance storm.
• Serializers: Avro/Protobuf + Schema Registry — smaller payloads than JSON at scale.

Security considerations

Kafka security for Java clients:

• SASL/SCRAM or mTLS: Authenticate producers/consumers — never plaintext in production.
• ACLs: Order service WRITE on order.events only — principle of least privilege.
• No PII in keys: Keys logged in metrics — use opaque order ID not email.
• Encryption at rest: Broker disk encryption + TLS in transit.

Scalability considerations

Scaling Kafka consumers and producers:

• Horizontal consumers: Add pods until consumer count = partition count — then add partitions (requires planning).
• Multiple consumer groups: Inventory and analytics both read order.events — independent scale.
• Broker scale: Add brokers + rebalance partitions — use cruise control for automation.
• Compacted topics: changelog topics for Kafka Streams state — retention by key, not time.

Production challenges

Real Kafka production issues:

• Consumer lag: Slow handler or insufficient consumers — scale or optimize processing.
• Rebalance storm: Processing exceeds max.poll.interval.ms — consumer kicked, partitions reshuffled, duplicate processing.
• Hot partitions: Null key or skewed key distribution — one partition overloaded.
• Poison messages: Deserialization failure — DLQ pattern with @RetryableTopic (Spring Kafka 2.7+).
• Schema evolution: Avro backward compatibility — new fields optional, never remove required fields.

Common mistakes

• Auto-commit before processing completes — message lost on crash after commit.
• More consumers than partitions — idle consumers waste resources; no speedup.
• No message key — round-robin partitioning loses per-entity ordering.
• Ignoring consumer lag alerts — backlog grows until retention deletes unprocessed messages.
• Exactly-once everywhere — unnecessary latency; at-least-once + idempotent handler often sufficient.

Debugging guide

Debug Kafka issues in Java services:

• Consumer lag: kafka-consumer-groups.sh --describe --group inventory-workers.
• Log correlation: Enable logging.level.org.apache.kafka=INFO — watch rebalance events.
• Trace propagation: Inject traceparent in record headers — link producer span to consumer span.
• Stuck consumer: Check max.poll.interval.ms vs handler duration — thread dump on consumer pod.

# Consumer group lag
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group inventory-workers

# Inspect topic messages
kafka-console-consumer.sh --bootstrap-server kafka:9092 \
  --topic order.events --from-beginning --max-messages 5

# List partitions and leaders
kafka-topics.sh --describe --topic order.events --bootstrap-server kafka:9092

Best practices

• Use message keys for entities needing ordered processing (orderId, userId).
• Set acks=all and min.insync.replicas=2 for durability.
• Enable idempotent producer — free dedup on retries.
• Manual commit or transactional listener after side effects succeed.
• Monitor consumer lag per group — primary health metric.
• Use Schema Registry (Avro) for production topics — not raw JSON.
• Implement DLQ for poison messages — don't block partition indefinitely.

Anti-patterns

• Using Kafka as job queue without keys — ordering and fairness break down.
• Giant messages (>1MB) — use S3 reference + event with pointer; Kafka for metadata.
• Single partition topic for high throughput — bottleneck; no consumer parallelism.
• Sync blocking send in request thread — use async callback or separate executor.
• Consumer doing heavy work in poll thread — blocks heartbeat; use async processing with pause/resume.

Staff engineer notes

• Staff engineers default to at-least-once + idempotent consumer — exactly-once only when financial correctness demands it.
• Partition count is forever-ish — changing it reorders keys; plan 2–3× expected peak consumer count.
• Consumer group ID is a deployment identity — never reuse group ID for different logic (reads wrong offsets).
• Kafka is not a database — retention expires data; use compacted topics or external store for source of truth.
• Spring @KafkaListener is production-ready — but understand what it wraps: poll loop, ack, rebalance listener.

Interview questions

Hands-on exercise

Lab: Kafka concepts in Java:

• Run playground — observe partition assignment simulation output.
• Add PaymentFailed event to sealed hierarchy — update handler switch.
• Write pseudo @KafkaListener method with manual ack comment.
• Calculate: 6 partitions, 4 consumers — how many idle consumers?
• Bonus: explain when you'd choose exactly-once vs at-least-once for payment events.

Press Run to execute. Real javac + JVM via remote API. Falls back to simulator on network failure.

Architecture trade-offs

• At-least-once vs exactly-once: At-least-once wins simplicity; exactly-once wins financial correctness at latency cost.
• Kafka vs RabbitMQ: Kafka wins throughput and replay; RabbitMQ wins routing flexibility and low-latency task queues.
• JSON vs Avro: JSON wins dev speed; Avro wins payload size and schema governance.
• Choreography vs orchestration: Kafka events enable choreography; orchestrator adds visibility at coupling cost.

Summary

Kafka is the nervous system of Java microservices — master producers, consumers, partitions, consumer groups, and exactly-once before wiring Spring @KafkaListener in production. Plan partition count early, key your messages, and treat consumer lag as a page-worthy alert.

Key takeaways

• Topics + partitions = parallelism; keys preserve per-entity ordering.
• Consumer groups scale consumption — max one consumer per partition per group.
• Idempotent producer + manual ack = production baseline.
• Exactly-once for money paths; at-least-once + idempotent handler elsewhere.
• Monitor consumer lag — primary Kafka health metric.