Consumer API

Introduction

What is Consumer API? The Consumer API polls records, processes them, and commits offsets.

Understanding the topic

What happens — Consumer API:

• Create KafkaConsumer with group.id.
• Subscribe to topic list.
• Loop: poll → process → commit.
• Close consumer on shutdown.

Visual explanation

Pipeline view:

Consumer polls records from topic partitions
  ↓
Process business logic (DB, API, etc.)
  ↓
Commit offset after success
  ↓
Monitor consumer lag

Step-by-step explanation

1. Subscribe — Consumer joins a group and receives partition assignments.
2. Poll — Fetch records in batches with consumer.poll().
3. Process — Run business logic for each record.
4. Commit — Save offset after successful side effects.
5. Repeat — Continue polling; rebalance if group membership changes.

Informative example

Example:

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-service");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("payment-events"));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            process(record);
        }
        consumer.commitSync();
    }
}

Execution workflow

Best practices

• Use manual commits for critical side effects.
• Make consumers idempotent with event IDs or unique constraints.
• Keep processing under max.poll.interval.ms or use pause/resume patterns.
• Send poison messages to DLT with failure metadata.

Common mistakes

• Committing before database writes complete.
• Scaling consumers beyond partition count and expecting more throughput.
• Blocking the poll loop with slow downstream calls.

Summary

Consumer API — Commit offsets only after successful processing; make handlers idempotent; monitor lag per partition.