Event Streaming

Introduction

Event streaming treats the continuous flow of domain events as a first-class data product — not a side effect of CRUD APIs. Netflix's Keystone pipeline processes trillions of events daily for recommendations, billing, and operational analytics.

Staff architects design stream topologies: source → enrich → branch → sink, with schema evolution, late-arriving data handling, and stream-table duality for stateful processing.

Real production story

Netflix's recommendation engine once polled batch ETL tables updated hourly — users who binge-watched a new series waited until the next day for personalized thumbnails. Moving to real-time event streaming via Keystone cut recommendation refresh latency from hours to under 30 seconds. A 2020 schema change in viewing events broke three downstream Flink jobs silently until a data quality monitor caught null genre fields — proving that streaming architecture needs contract tests, not just broker uptime.

Business problem

Business pressure: Netflix personalization, A/B experiments, and content licensing analytics require sub-minute event freshness across 200M+ members. Batch pipelines cannot support interactive product features or real-time operational dashboards.

• Revenue at risk: Stale recommendations reduce watch time — directly tied to subscription retention and content ROI.
• Engineering velocity: Batch ETL contracts block data science teams; streaming enables self-service event subscriptions.
• Compliance / trust: Viewing and billing events need lineage tracking for content partner royalty calculations.

Architecture overview

Event streaming is the practice of capturing, transporting, processing, and sinking continuous event flows. It combines broker infrastructure (Kafka), stream processors (Flink, Kafka Streams), and schema governance into a cohesive data movement architecture.

• Definition: End-to-end pipeline treating events as durable, replayable streams with real-time processing capability.
• When to adopt: Product features need sub-minute freshness; multiple teams consume same events in real-time and batch.
• When to defer: Daily/hourly freshness suffices and volume is low — scheduled ETL is cheaper.
• Operability: End-to-end latency, schema compatibility, checkpoint health, and data quality metrics per stream.

Architecture motivation

Why architects care: Event streaming unifies real-time and batch — the same Kafka topic feeds live Flink jobs and nightly Spark jobs via tiered retention. The naive alternative — separate real-time and batch pipelines — doubles maintenance and guarantees schema drift.

• Force: Continuous high-volume events with multiple real-time and batch consumers.
• Constraint: Cannot replay entire history on every schema change — need compatible evolution.
• Outcome: Keystone-style platform with schema registry, stream processing templates, and data quality monitors.

Internal architecture

Netflix Keystone streaming topology — multi-tenant ingestion with stream processing:

• Keystone standardizes ingest — producers never write raw to Kafka directly.
• Flink checkpointing to S3 every 60s — recovery RPO < 2 minutes.
• Same topic serves real-time and batch via Connect sink — single source of truth.

Viewing Service → Keystone Ingest API
   ↓ Avro + schema_id
Kafka: viewing.events.v3 (512 partitions)
   ↓
├─ Flink: real-time genre affinity (RocksDB state, 1m window)
├─ Flink: A/B experiment metrics (session windows)
├─ Kafka Connect → S3 (Iceberg, hourly compaction)
└─ Kafka Streams: device anomaly detection

Data Quality Monitor:
  - null_rate(genre_id) < 0.01%
  - event_count per minute within 3σ of baseline
  - schema_id always registered

Data flow

Ingest: service → Keystone API → schema validate → Kafka append. Process: Flink reads changelog, maintains keyed state, emits derived events. Sink: Connect or custom sink writes to Iceberg, Redis, or downstream Kafka topics.

• Write path: HTTP/gRPC ingest with schema_id → broker append with member_id partition key.
• Read path: Flink source with event-time watermarks; allowed lateness for out-of-order events.
• Async path: Derived events (genre_affinity_updated) published to downstream topic for recommendation service.

// Flink streaming job — viewing event enrichment (Java)
DataStream<ViewingEvent> views = env
  .addSource(new FlinkKafkaConsumer<>("viewing.events.v3", schema, props))
  .assignTimestampsAndWatermarks(
    WatermarkStrategy.<ViewingEvent>forBoundedOutOfOrderness(Duration.ofSeconds(30))
      .withTimestampAssigner((e, ts) -> e.getTimestamp())
  );

DataStream<GenreAffinity> affinity = views
  .keyBy(ViewingEvent::getMemberId)
  .window(TumblingEventTimeWindows.of(Time.minutes(5)))
  .aggregate(new GenreCountAggregator())
  .name("genre-affinity-5m");

affinity
  .addSink(new FlinkKafkaProducer<>("recommendations.genre-affinity.v1", schema, props))
  .name("sink-genre-affinity");

env.enableCheckpointing(60_000);
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);

System design diagram

Two diagrams show the Event Streaming topology and the primary request/event path used in production at scale.

Production code example

Stream job deployment template — Netflix platform standard:

• Platform owns checkpoint config — product teams do not tune blindly.
• Data quality rules deployed with job — fail job on contract breach.
• Savepoint-before-deploy mandatory for stateful job upgrades.

# flink-job.yaml
apiVersion: streaming.netflix/v1
kind: FlinkJob
metadata:
  name: genre-affinity-v2
spec:
  source:
    topic: viewing.events.v3
    consumerGroup: genre-affinity-v2
  sink:
    topic: recommendations.genre-affinity.v1
  parallelism: 128
  checkpoint:
    intervalMs: 60000
    mode: EXACTLY_ONCE
    stateBackend: rocksdb
  watermarks:
    maxOutOfOrderSeconds: 30
  dataQuality:
    rules:
      - field: genre_id
        nullRateMax: 0.0001
      - metric: events_per_minute
        deviationSigma: 3

Enterprise case study

Netflix — Keystone event streaming platform: Unified ingest, schema registry, Flink job templates, and data quality monitors. Product teams deploy streaming jobs via self-service with platform-enforced checkpoint and state backend configs.

• Before: Hourly batch ETL for recommendations; schema drift between real-time and batch paths.
• Decision: Keystone as mandatory ingest; Flink for real-time; Iceberg sink for batch convergence.
• After: Recommendation refresh < 30s; data quality monitors caught 12 schema issues pre-production in first year.

Trade-offs

• Event time vs processing time: Event time handles out-of-order correctly; processing time is simpler but wrong under lag.
• Stateful vs stateless: Stateful Flink jobs need checkpoint storage and recovery drills; stateless is easier but limited.
• Lambda vs Kappa: Kappa (stream-only) simplifies; Lambda (batch+stream) needed when batch algorithms differ materially.
• At-least-once vs exactly-once: Exactly-once in Flink adds latency; many sinks achieve effectively-once via idempotent writes.

Security considerations

Viewing data is PII: Stream pipelines need encryption, access controls per topic, and audit on schema changes.

• Identity: Service accounts per Flink job with consume-only on source, produce-only on sink topics.
• Data: Hash member_id in analytics sinks; raw ID only in authorized recommendation pipelines.
• Supply chain: Pin Flink and connector versions; test job upgrades with saved savepoint recovery.

Scalability analysis

Scale dimensions: Netflix processes trillions of events daily. Kafka partition count, Flink parallelism, and RocksDB state size are the bottlenecks.

• Horizontal scale: Flink parallelism = Kafka partition count for 1:1 mapping; scale both together.
• Hot spots: Viral content spikes viewing events — auto-scale Flink task managers on lag SLO breach.
• Cost: Checkpoint storage and retained Kafka volume dominate — tier to S3 aggressively.

Failure scenarios

What breaks: Schema drift breaks Flink deserialization; checkpoint failure loops job restart; watermark stall from silent partition.

• Schema break: New required field without default — enforce BACKWARD compatibility in registry.
• Checkpoint timeout: State too large — increase interval, tune RocksDB, or redesign state.
• Silent partition: One partition stops receiving — watermark does not advance, windows never fire — alert on per-partition rate.

Staff engineer insights

• Event streaming is a platform capability — not a Kafka topic and a cron job.
• Data quality monitors on streams matter more than broker uptime alerts.
• Watermarks and allowed lateness are design decisions — document them per job.
• The same topic feeding real-time and batch (Kappa) eliminates the biggest source of data drift.

Interview questions

Architecture review questions

• Schema compatibility mode set and contract tests in CI?
• Event-time watermarks and allowed lateness documented?
• Checkpoint interval and state backend sized with recovery drill evidence?
• Data quality monitors on critical fields (null rate, volume)?
• Per-partition throughput alerts to catch silent partitions?
• Same source topic serves both real-time and batch consumers?

Summary

Event streaming at Netflix scale means Keystone-style ingest, Flink processing with event-time semantics, and data quality monitors that catch schema drift before customers do. Design streams as data products with contracts — not as log tail afterthoughts.