Streams

Introduction

Streams are append-only logs with auto IDs — XADD appends events; consumer groups (XREADGROUP) track progress like lightweight Kafka inside Redis. LinkedIn-style feed pipelines and order event sourcing use streams when pub/sub loses messages.

Each entry has fields (key-value pairs). Pending Entry List (PEL) tracks in-flight messages until XACK. XCLAIM reassigns stale pending from crashed consumers.

Trim with MAXLEN ~ approximate cap to bound memory on high-volume streams.

Understanding the topic

Key concepts

• XADD stream * field value — auto timestamp ID.
• XREAD BLOCK — tail new entries.
• XGROUP CREATE — consumer group on stream.
• XREADGROUP GROUP g consumer — claim new messages.
• XACK — mark processed; removes from PEL.
• XPENDING / XCLAIM — recover stuck messages.

flowchart LR
  Producer -->|XADD| Stream
  Stream -->|XREADGROUP| Consumer

Step-by-step explanation

1. Stream stored as radix tree of listpacks by ID.
2. Consumer group maintains last-delivered ID and PEL.
3. New message delivered to one consumer in group.
4. Unacked messages stay pending until XACK or XCLAIM.
5. XTRIM or MAXLEN on XADD caps stream length.

Syntax reference

Common commands

• > — only new undelivered to group.
• $ — start from end on group create.
• MKSTREAM creates stream if missing.

XADD orders * type OrderCreated id 12345
XGROUP CREATE orders processors $ MKSTREAM
XREADGROUP GROUP processors c1 COUNT 10 STREAMS orders >
XACK orders processors 1690000000000-0

Informative example

Producer adds order events; consumer group processes with ack:

XADD events:orders * orderId 99 amount 42.50 status CREATED
XREADGROUP GROUP fulfillment worker-1 COUNT 1 BLOCK 5000 STREAMS events:orders >
# process...
XACK events:orders fulfillment <message-id>

On worker crash, XPENDING + XCLAIM after idle timeout reassigns work.

Real-world use

Real-world use cases

• Order lifecycle event log.
• Activity feed ingestion buffer.
• Microservice outbox pattern drain.
• Sensor telemetry batch ingest.
• Audit trail with trim policy.

Best practices

• Always XACK after successful processing.
• Set MAXLEN ~ on XADD for bounded memory.
• Monitor XPENDING length per group.
• Idempotent consumers — redelivery happens.
• Separate stream per domain: events:orders.
• XCLAIM idle time > p99 processing time.

Common mistakes

• Never XACK — PEL grows forever.
• Unbounded stream without trim.
• Non-idempotent handlers on redelivery.
• Single consumer group doing unrelated work.

Advanced interview questions

Summary

Streams = durable append log with consumer groups.