Kafka

Introduction

Apache Kafka is a distributed commit log — partitions, replicas, consumer groups, and ISR management form the core architecture LinkedIn pioneered for activity streaming. Staff engineers design around log semantics: append-only, ordered per partition, replayable, and retained by policy.

This lesson covers broker cluster topology, partition/replica placement, consumer group rebalancing, exactly-once producer configs, and how LinkedIn's original use case — unified activity pipeline — still informs modern event streaming design.

Real production story

LinkedIn's Kafka journey began when multiple teams built bespoke ingestion pipelines for profile views, job applications, and feed updates — each with different SLAs and failure modes. Consolidating into a single log-based bus cut ingestion latency variance from minutes to seconds and let new product teams subscribe to events without negotiating with every upstream owner. When a 2022 broker rack misconfiguration dropped ISR below min.insync.replicas, the activity pipeline paused writes for 8 minutes — teaching the org that Kafka architecture is replication math, not just topic names.

Business problem

Business pressure: LinkedIn's real-time features — feed ranking, notifications, analytics — depend on a unified event log with predictable latency and replay capability. Fragmented pipelines block product velocity and create inconsistent member experiences.

• Revenue at risk: Delayed feed updates and missed notifications reduce engagement metrics tied to ad revenue.
• Engineering velocity: Without a shared log, every new feature negotiates bespoke ingestion contracts.
• Compliance / trust: Member activity data requires retention policies and access controls auditable per topic.

Architecture overview

Kafka architecture centers on topics partitioned across brokers, each partition replicated for fault tolerance. Producers append to partition leaders; followers replicate; consumers read via pull-based consumer groups that track offsets.

• Definition: Distributed, partitioned, replicated commit log with consumer-group based consumption.
• When to adopt: Event streaming, log aggregation, stream processing, or when replay is a first-class requirement.
• When to defer: Low-volume task queues with no replay need — SQS is simpler and cheaper.
• Operability: Monitor under-replicated partitions, ISR shrink events, consumer lag per partition, and disk usage.

Architecture motivation

Why architects care: Kafka's log abstraction solves fan-out, replay, and backpressure simultaneously. The naive alternative — point-to-point queues per consumer — cannot replay historical events for new consumers or recover from consumer bugs without expensive backfill jobs.

• Force: High-throughput, multi-subscriber event streams with retention measured in days to years.
• Constraint: Cannot lose ordering for member-scoped events (profile, connections).
• Outcome: Tiered topics, rack-aware replication, and consumer group standards documented in platform ADRs.

Internal architecture

LinkedIn-scale Kafka cluster — rack-aware, tiered storage:

• Rack-aware replica assignment survives single-rack failure without leader election storm.
• Partition count set at topic creation — increasing later requires careful rebalancing.
• Tiered storage moves cold segments to object store; hot path stays on NVMe.

Topic: member.activity.v1
  partitions: 128
  replication.factor: 3
  min.insync.replicas: 2
  retention.ms: 604800000 (7d hot)
  → Tiered to S3 after 24h (cold)

Broker rack placement:
  broker-1 (rack-a) ← leader P0
  broker-2 (rack-b) ← follower P0
  broker-3 (rack-c) ← follower P0

Consumer group: feed-ranker-v4
  max.poll.records: 500
  session.timeout.ms: 45000
  isolation.level: read_committed

Data flow

Produce: producer hashes key → partition leader appends → followers replicate → ack per acks config. Consume: consumer polls batch → processes → commits offset synchronously or via transactional producer.

• Write path: key=member_id ensures per-member ordering; idempotent producer prevents duplicate sequence numbers.
• Read path: Consumer group coordinator assigns partitions; rebalance on member join/leave.
• Async path: Kafka Streams / Flink reads changelog topics for stateful processing.

// Production Kafka producer config (Java)
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put("schema.registry.url", registryUrl);

// Consumer with manual commit after processing
while (true) {
  ConsumerRecords<String, ActivityEvent> records = consumer.poll(Duration.ofMillis(500));
  for (ConsumerRecord<String, ActivityEvent> record : records) {
    processActivity(record.value());
  }
  consumer.commitSync(); // only after batch processed
}

System design diagram

Two diagrams show the Kafka topology and the primary request/event path used in production at scale.

Production code example

Kafka topic provisioning — platform-as-code with guardrails:

• Declare producers/consumers at provision time — ACLs generated automatically.
• Schema compatibility enforced before topic goes live.
• Platform team owns broker health; product teams own consumer lag SLOs.

# topic-provision.yaml — LinkedIn platform standard
apiVersion: kafka.platform/v1
kind: Topic
metadata:
  name: member.activity.v1
spec:
  partitions: 128
  replicationFactor: 3
  config:
    min.insync.replicas: "2"
    retention.ms: "604800000"
    compression.type: "lz4"
    cleanup.policy: "delete"
  access:
    producers: [activity-ingestor]
    consumers: [feed-ranker, notification-svc, analytics-lake]
  schema:
    subject: member.activity.v1-value
    compatibility: BACKWARD

Enterprise case study

LinkedIn — unified activity pipeline on Kafka: All member-facing events flow through tiered topics with schema registry governance. New product teams onboard via self-service topic provisioning with default replication and retention policies.

• Before: 30+ bespoke ingestion pipelines, inconsistent SLAs, no replay for new consumers.
• Decision: Kafka as system of record for activity events; deprecate legacy pipelines over 18 months.
• After: New consumer onboarding dropped from weeks to days; replay enabled analytics backfill without source system load.

Trade-offs

• acks=all vs acks=1: All waits for ISR replication — safer, higher latency; 1 is faster but risks loss on leader crash.
• Partition count: More partitions = more parallelism but more file handles and rebalance overhead.
• KRaft vs ZooKeeper: KRaft simplifies ops; migration path matters for existing clusters.
• Compaction vs retention: Compacted topics for changelog; time-retained for event streams — mixing semantics causes confusion.

Security considerations

Kafka carries PII: Member activity events require encryption, ACLs per topic, and audit logging for admin operations.

• Identity: SASL/SCRAM or mTLS per service principal; ACLs limit produce/consume to declared topics.
• Data: Encrypt at rest on broker disks; consider field-level encryption for sensitive attributes.
• Supply chain: Pin broker and client versions; test upgrades in staging with production traffic replay.

Scalability analysis

Scale dimensions: LinkedIn Kafka clusters handle trillions of messages. Broker disk bandwidth, partition count, and consumer fetch size dominate — not producer CPU.

• Horizontal scale: Add brokers + rebalance partitions; consumers scale up to partition count.
• Hot spots: Celebrity member activity floods one partition — consider sub-key salting for analytics-only consumers.
• Cost: Tiered storage and retention policies are the primary cost levers — not broker instance size alone.

Failure scenarios

What breaks: ISR shrink below min.insync.replicas blocks producers; consumer rebalance storm during deploy; disk full on broker.

• Under-replicated partitions: Follower falls behind — alert immediately; may indicate network or disk issue.
• Rebalance storm: Frequent consumer join/leave during rolling deploy — use static membership or cooperative rebalancer.
• Disk exhaustion: Retention without tiered storage fills broker — automated alerts at 70% disk.

Staff engineer insights

• Partition count is a capacity plan decision — you cannot casually double it in production.
• min.insync.replicas + acks=all is the default for anything member-facing — justify exceptions in an ADR.
• Consumer lag per partition, not just aggregate — a single hot partition hides in averages.
• Kafka is a log, not a queue — design consumers to tolerate replay and duplicates.

Interview questions

Architecture review questions

• replication.factor ≥ 3 and min.insync.replicas ≥ 2 for production topics?
• Partition key strategy documented with hot-partition mitigation plan?
• Consumer group rebalance strategy chosen (cooperative vs eager)?
• Tiered storage or retention policy matches compliance requirements?
• Under-replicated partition alerts wired to on-call?
• Schema registry compatibility mode set per topic?

Summary

Kafka architecture at LinkedIn scale means rack-aware replication, deliberate partition planning, and consumer groups designed for rebalance tolerance. Master ISR semantics, tiered retention, and idempotent consumption — that is staff-level Kafka, not topic creation tutorials.