Tenant Isolation

Introduction

Tenant Isolation is the cross-cutting quality attribute — not a storage model — ensuring one tenant's load, failures, and security boundaries cannot impact another. Netflix applies isolation at compute (bulkheads), queue (per-tenant topics), network (security groups), and data layers. Isolation is measured, not assumed.

Real production story

Netflix's partner encoding platform shared one Kafka topic and one GPU worker pool for all studios. When Disney's holiday catalog dump flooded the queue with 40K 4K jobs, Universal's time-sensitive episodic deliverables stalled 90 minutes — contractual SLA breach and executive escalation.

The staff architect defined isolation as SLO: each studio gets dedicated queue partition set, GPU bulkhead (min 20% capacity reserved per tier-1 studio), rate limits on job submission, and circuit breaker when one studio's failure rate exceeds threshold. Isolation dashboard shows per-studio latency, queue depth, and blast-radius score. Shared infrastructure remained — isolation moved to resource governance, not just database models.

Business problem

Business pressure: Netflix studio partners pay for guaranteed delivery windows — shared infrastructure without isolation governance treats all partners as equal until one floods the system.

• Revenue at risk: SLA penalties and partner churn when one studio's traffic impacts another's deliverables.
• Engineering velocity: Platform team ships shared services — product requirement is isolation guarantees on shared infra.
• Compliance / trust: Content security requires studio A cannot access studio B's assets — isolation spans compute, network, and data.

Architecture overview

Tenant Isolation in production spans: data isolation (tenant_id, RLS, separate DB), compute isolation (bulkheads, quotas), network isolation (security groups, mTLS per tenant context), and failure isolation (circuit breakers, per-tenant timeouts). Measure each dimension — don't conflate shared database with zero isolation.

• Definition: Quality attribute ensuring one tenant's operations, failures, and data cannot violate another tenant's SLO or security boundary.
• When to adopt: Always in multi-tenant systems — isolation level scales with tenant tier and contract.
• When to defer: Never defer isolation design — defer only expensive isolation tiers (separate DB) until contracts require.
• Operability: Per-tenant dashboards: latency, error rate, queue depth, capacity share, cross-tenant access audit.

Architecture motivation

Why architects care: Tenant isolation is the quality attribute that makes any multi-tenant model (shared DB, shared schema, separate schema, separate DB) trustworthy. Without measured isolation, storage model choice is cosmetic.

• Force: Shared infrastructure economics with enterprise isolation expectations.
• Constraint: Cannot dedicate full stack per studio at Netflix partner count — need logical isolation on shared fleet.
• Outcome: Isolation SLOs per tenant: max queue wait, reserved capacity, failure blast radius metrics.

Internal architecture

Netflix partner platform isolation layers:

Layer 1 — Data:     studio_id on all assets; S3 prefix per studio; IAM scoped
Layer 2 — Queue:      Kafka topic partition by studio_id; max inflight per studio
Layer 3 — Compute:    GPU pool bulkheads — tier-1 studios min 20% reserved
Layer 4 — Network:    VPC security groups; mTLS with studio claim in cert
Layer 5 — Failure:    circuit breaker per studio; timeout isolation
Layer 6 — Observability: per-studio RED metrics + blast-radius score

Data flow

Encode job submission: studio authenticates → rate limiter checks studio quota → job routed to studio's partition → GPU worker from studio's bulkhead pool picks job → metrics emitted per studio_id.

• Write path: asset upload to s3://bucket/{studio_id}/ — IAM denies cross-prefix access.
• Read path: encode status API filters by authenticated studio — never list all jobs.
• Async path: per-studio DLQ for failed encodes — one studio's poison messages don't block others.

class StudioIsolationGuard {
  constructor(
    private rateLimiter: RateLimiter,
    private bulkhead: BulkheadRegistry,
    private circuitBreaker: CircuitBreakerRegistry,
  ) {}

  async submitJob(studioId: string, job: EncodeJob): Promise<void> {
    await this.rateLimiter.acquire(studioId, { maxPerMinute: tierQuota(studioId) });
    if (this.circuitBreaker.isOpen(studioId)) {
      throw new StudioCircuitOpenError(studioId);
    }
    const pool = this.bulkhead.getPool(studioId);
    const partition = hashStudio(studioId) % PARTITION_COUNT;
    await kafka.publish("encode.jobs", job, { partition, headers: { studioId } });
    metrics.increment("encode.job.submitted", { studioId });
  }
}

System design diagram

Two diagrams show the Tenant Isolation topology and the primary request/event path used in production at scale.

Production code example

Isolation game day test — synthetic neighbor load in staging:

// game-day/isolation-flood.test.ts
describe("tenant isolation under neighbor flood", () => {
  it("studio B p99 stays under SLA when studio A floods queue", async () => {
    const flood = floodStudio("studio-a", { jobsPerSec: 500, durationSec: 60 });
    const probe = measureLatency("studio-b", { jobsPerSec: 10 });
    const [_, bMetrics] = await Promise.all([flood, probe]);
    expect(bMetrics.p99Ms).toBeLessThan(STUDIO_B_SLA_MS);
  });
});

Enterprise case study

Netflix partner encoding — post-90-minute incident: Implemented bulkhead GPU pools, per-studio Kafka partitions, rate limits, circuit breakers, and isolation dashboard reviewed weekly with partner ops.

• Before: 90-minute Universal delay from Disney queue flood; SLA penalties.
• Decision: Isolation as measured SLO on shared infra — not immediate separate infrastructure per studio.
• After: Zero cross-studio SLA breaches in 24 months; tier-1 studios pay isolation premium.

Trade-offs

• Shared infra vs isolation overhead: Bulkheads and quotas reduce utilization — buy partner SLA compliance.
• Strict vs fair scheduling: Tier-1 reserved capacity starves tier-3 during peak — document fairness policy.
• Measurement cost vs blind spots: Per-tenant metrics cardinality expensive — sample tier-3, full tier-1.

Security considerations

Security is architectural: Tenant isolation includes preventing studio A from reading studio B's content — data path isolation is security requirement, not just performance.

• Identity: studio_id in mTLS cert and JWT; every API validates scope.
• Data: S3 prefix IAM; cross-studio access requires break-glass with audit.
• Supply chain: encode worker containers cannot mount other studios' storage paths.

Scalability analysis

Scale dimensions: Netflix partners range from indie studios to Disney — isolation policy must tier without exploding configuration cardinality.

• Horizontal scale: bulkhead pools scale independently; partition count grows with studio count.
• Hot spots: major studio release day — pre-allocated capacity surge request 48h ahead.
• Cost: reserved capacity is idle cost — charge tier-1 studios for isolation premium.

Failure scenarios

What breaks: bulkhead misconfiguration shares pool; rate limiter bypass in admin API; circuit breaker stuck open blocks studio indefinitely.

• Shared bulkhead bug: integration test submits concurrent jobs from two studios, asserts latency independence.
• Admin bypass: admin APIs subject to same isolation guards with audit log.
• Stuck circuit: half-open probe after cooldown; manual reset requires ticket.

Staff engineer insights

• Isolation is a quality attribute with metrics — if you cannot show per-tenant latency under neighbor load, you don't have isolation.
• Storage model (shared DB vs separate DB) is one layer — compute and queue isolation matter equally for Netflix workloads.
• Game days: flood one tenant synthetically, verify others meet SLO — isolation tests belong in CI/CD.

Interview questions

Architecture review questions

• Are quality attributes (latency, availability, consistency) explicit with SLOs for Tenant Isolation?
• Is the failure/degraded mode documented — including what happens when dependencies are down?
• Are boundaries and ownership clear on an architecture diagram a new engineer understands in 10 minutes?
• Is there an ADR capturing alternatives considered and why they were rejected?
• Can this design scale 10× on traffic and 3× on engineering headcount without a rewrite?
• Security: authn/authz, encryption, and blast radius reviewed at every external interface?

Summary

Tenant Isolation at Netflix scale is a measurable quality attribute — bulkheads, partitioned queues, rate limits, and circuit breakers on shared infrastructure, complemented by data-layer controls. Design isolation tests, dashboard per tenant, and tier isolation premium into enterprise contracts.