Design a Multi-Tenant SaaS Platform

Advanced System Design & Software Architecture · Lesson 60 · Real-World System Design Case Studies

Lesson Overview

TechLearningPro Cloud hosts a public catalog and thousands of universities and companies on one product. A learner at Northwind University must never read Eastbridge College's roster. A Monday-morning attendance import at the largest campus must not push sign-in latency for every other tenant past its SLO. And finance must be able to say, at the end of the month, how many active learners and how many gigabyte-hours each tenant consumed, once, without a spreadsheet that re-counts retries.

That is multi-tenant SaaS. Tenancy is the rule that every request, row, cache key, job, and bill belongs to exactly one tenant, and that shared infrastructure does not become shared data. The architecture choice is how much hardware and how much schema each tenant shares. Microsoft's Azure architecture guidance names three useful postures: silo (dedicated resources per tenant), bridge (shared application tier, isolated data store), and pool (shared application and shared tables discriminated by tenant_id). This lesson uses those names in that sense and points at the current guide rather than inventing a fourth. TechLearningPro's default is pool with PostgreSQL row-level security as a backstop, bridge (a dedicated database) for tenants who pay for stronger isolation, and silo (a dedicated deployment) only when a contract requires it. Most tenants will never justify silo. The design has to make pool safe enough that we are not forced there by fear.

Identity is resolved from the host (northwind.cloud.techlearningpro.example) and must match the tenant claim in the token. A tenant id in a JSON body is not authority. Data access goes through repositories that set app.tenant_id for the transaction and rely on row-level security so a forgotten WHERE is not the only barrier. Noisy neighbors are limited with per-tenant rate limits, statement timeouts, and fair queues for heavy work such as imports and exports. Billing is an append-only meter stream with idempotency keys, rolled up daily, not a COUNT(*) on the live roster at invoice time.

Lessons 55 through 59 are features that run on this platform: payments, notifications, feeds, files, and dispatch. Each already mentioned tenant_id. This lesson is the platform contract they all inherit. Module 6's capstone will compose those features for a new tenant type. Enterprise Architecture Review Checklist will ask you to show isolation tests, quota behavior, and the migration story for a shared schema. Architecture Security and Threat Modeling will ask what a single missing predicate leaks.

Learning Objectives

By the end of this lesson, you should be able to:

1. Define a tenant, a principal, and the control plane versus the data plane.
2. Compare silo, bridge, and pool isolation, including cost, noisy-neighbor blast radius, and operational load.
3. Resolve tenant identity from the host and the token, and reject a body-supplied tenant id as authority.
4. Apply PostgreSQL row-level security with SET LOCAL, and explain why a session-level SET leaks across a pool.
5. Specify quotas, rate limits, and fair queues that keep one tenant's import off another's sign-in path.
6. Design append-only usage meters and daily rollups that survive retries without double billing.
7. Place cache keys, search filters, and background jobs so tenant context cannot be dropped.
8. Plan schema migrations for a pooled database where one migration touches every tenant.
9. Offer customer-managed keys as envelope encryption without a second copy of plaintext "for support."
10. Estimate a long-tail tenant population and say which tier a large tenant can move to.
11. Review a design that filters tenant id only in the UI.
12. Carry the platform into the Module 6 capstone as the isolation and billing baseline.

Prerequisites

• Lesson 30 and lesson 31: a shared table is a partitioning and consistency choice. tenant_id is the leading key in pooled indexes.
• Lesson 35: per-tenant token buckets are the first noisy-neighbor control.
• Lesson 37: meter events and invoice adjustments are idempotent commands.
• Lesson 40: APIs are versioned once for all pool tenants. A breaking change is a platform event.
• Lesson 44: service credentials carry the tenant they may act for. A notification worker is not a superuser that skips policy.
• Lesson 48: expand-contract migrations. Pooled schemas cannot take a long exclusive lock because every tenant feels it.
• Lessons 55–59: payment ledgers, notification audiences, feed visibility, object keys, and geo indexes must all include the tenant. This lesson is why.

Detailed Core Concepts

What is being isolated

A tenant is the customer organization: a university, a company, or the platform's own public catalog, which is a tenant like the others so the first-party product does not get a back door. A principal is a person or a service identity inside a tenant. Roles (learner, instructor, admin) are data in that tenant, not separate deployments. The control plane creates tenants, plans, keys, and quotas. The data plane serves learner and instructor traffic. Control-plane credentials do not ride along on a learner request. A support impersonation is a separate, audited, time-boxed token that still carries the target tenant id.

Isolation has three layers, and you need all three in the pool model:

Any single layer will be bypassed by a bug, a replica, a cache, or a new service. Defense in depth here is not a slogan. It is how a missing WHERE clause becomes a failed test instead of a breach.

Silo, bridge, and pool

These are deployment postures. They can be mixed per tier (compute pooled, data bridged).

TechLearningPro's rule: start pool. Move a tenant to bridge when their sustained database load, residency, or contractual isolation requires a separate database. Move to silo when they require dedicated compute, a private network, or a change window they control. The application code is the same. The connection string and the Kubernetes namespace change. Do not fork the product per tier. A fork is how silo tenants fall a version behind and how security patches get "scheduled" forever.

Schema-per-tenant (a form of bridge) looks tidy and breaks connection pools, ORMs, and migration tooling in ways a separate database also breaks, with less clear resource isolation. This design prefers database-per-tenant when leaving the pool, not a hundred schemas in one server, unless the operational tooling already exists. Say that trade-off rather than treating "schema" and "database" as synonyms.

Tenant identity

Resolution order:

1. Map the HTTP host to a tenant id through a cached control-plane table. Unknown host is 404, not a default tenant.
2. Authenticate the caller. The access token contains tid. If tid does not equal the host's tenant, reject the request. A token minted for Eastbridge must not work on Northwind's host even if someone copies it.
3. Ignore tenant_id fields in bodies and query strings for authorization. They may be present for debugging and must match or be absent. They never override the host and token.

Background jobs have no host. The message body must contain the tenant id, and the worker sets the database context from the message, not from a thread-local leftover from the previous job. Thread-locals are how a pooled worker thread leaks tenant A into tenant B's import. Pass TenantId as a field. Set it at the start of the unit of work. Clear it in a finally if you use a thread-local at all. Prefer an explicit parameter on every repository method so a missing value does not compile.

Row-level security

PostgreSQL row-level security expresses the pool invariant in the database:

sqlCopy
ALTER TABLE enrollments ENABLE ROW LEVEL SECURITY;
ALTER TABLE enrollments FORCE ROW LEVEL SECURITY;

CREATE POLICY enrollments_tenant ON enrollments
  USING (tenant_id = current_setting('app.tenant_id', true)::uuid)
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true)::uuid);

FORCE applies the policy even to the table owner. Without it, a migration role and a misconfigured admin role bypass the policy and become the leak. The application role is not a superuser. Superuser and BYPASSRLS roles are forbidden on the request path.

At the start of each transaction the repository runs SELECT set_config('app.tenant_id', ?, true). The third argument true means the setting is local to the transaction, equivalent to SET LOCAL. A bare SET survives on a pooled connection and the next request inherits the previous tenant. That bug is quiet and catastrophic. Tests must check out a connection, set tenant A, return it, check it out for tenant B, and prove B cannot read A's row. current_setting(..., true) returns NULL when missing; the comparison then fails closed and the query sees no rows. Fail closed. A missing setting that disables the policy is the wrong default.

RLS is the backstop. The application still adds tenant_id to queries so planners use the index (tenant_id, ...) and so a policy mistake is visible in code review. RLS without the index predicate can sequential-scan. The index without RLS can leak. Use both.

Noisy neighbor

In a pool, one tenant can hurt others without reading their data:

The control you can implement this week is the rate limit plus the statement timeout plus a separate queue for imports. The control that requires product work is the fair queue. Name which one you have. "We will autoscale" does not protect a single primary database. Autoscaling stateless API pods spreads CPU and does nothing for a lock on plans.

Silo and bridge shrink the noisy-neighbor set by construction: their database is not shared. They still share a control plane, a CI system, and often a network egress. Do not claim infinite isolation.

Quotas and plans

A plan defines soft and hard quotas: active learners, storage gigabyte-hours, notification sends, dispatch jobs, API calls per day. Soft quota notifies tenant admins and does not fail the learner's request. Hard quota fails the specific action (402 or 429 with a problem type quota-exceeded) and does not fail unrelated actions. A storage hard quota blocks new uploads and still allows downloads and sign-in. A notification hard quota blocks marketing sends before it blocks security passcodes; that priority matches lesson 56.

Quotas are enforced in the service that owns the resource, using the rollup plus today's meter increments, not by asking an admin service on every request if that hop becomes a bottleneck. A short cache of "headroom remaining" is allowed if a hard quota check on the write path still cannot go negative by more than a small overshoot you document (for example one in-flight upload). Perfect quota under concurrency is a serial counter. Near-perfect with a small overshoot is the practical pool design.

Billing meters

The invoice is not a live query. Each billable action appends a usage event:

• tenant_id, meter (active_learner_day, storage_gb_hour, notification_send, api_call), quantity, occurred_at
• idempotency_key unique per tenant, so a retried request does not bill twice
• optional source_id (file version, delivery id) for audit

A daily job writes rollups (tenant_id, meter, day, quantity). Invoices read rollups. Late events land in the day they occurred and a correction rollup if that day was already invoiced, rather than silently mutating a closed invoice. Adjustments are new events with a reason and an actor. This is the same append-only habit as the payment ledger in lesson 55, at lower stakes and the same need for idempotency.

active_learner_day is "a distinct principal had a learning activity that day," emitted once per principal per day via an idempotency key learner:{id}:{date}. Emitting it on every page view and summing quantities would over-bill. The meter definition is part of the contract with finance. Engineers do not change the meaning of a meter in a patch release.

Migrations and the long tail

A pooled schema means a migration runs for everyone. Expand-contract from lesson 48: add a nullable column, backfill in batches by tenant so one tenant's million rows do not lock the table for all, switch reads, then drop the old shape. Avoid ACCESS EXCLUSIVE locks during the business morning of the largest tenant. Bridge tenants run the same migration code against their database on a fleet job with a concurrency cap, and a tenant can be paused. Silo tenants are in that fleet with a slower cadence if the contract says so, and they do not stay on a vulnerable version past the security window. "We cannot restart them" is not a patch policy.

Feature flags are tenant-scoped. A global flag that you flip because one tenant asked will change the public catalog. The flag service requires a tenant id, and the default is the platform default, not "whatever the last request set."

Architecture and Design Diagrams

The control plane is small and highly privileged. The data plane is large and tenant-scoped on every hop. Workers consume tenant id from the message.

mermaidCopy
flowchart LR
  subgraph edge [Edge]
    LB[L7 load balancer]
    Host[Host to tenant map]
  end

  subgraph data [Data plane]
    API[Tenant APIs]
    Workers[Import and export workers]
    DB[(Pooled PostgreSQL with RLS)]
    Cache[(Redis keys prefixed by tenant)]
  end

  subgraph control [Control plane]
    CP[Tenant admin API]
    CDB[(Control database plans and hosts)]
    Bill[Meter rollup]
  end

  User[Browser] --> LB --> Host --> API
  API --> DB
  API --> Cache
  API --> Workers
  Workers --> DB
  CP --> CDB
  API --> CDB
  Bill --> DB
  API --> Bill

mermaidCopy
sequenceDiagram
  participant B as Browser
  participant API as Tenant API
  participant DB as PostgreSQL
  participant M as Meter log

  B->>API: Host northwind plus token tid
  API->>API: Reject if host tenant and tid differ
  API->>DB: BEGIN; set_config app.tenant_id local
  API->>DB: Insert enrollment WITH CHECK policy
  API->>M: Usage event idempotency key
  API->>DB: COMMIT
  API-->>B: 201

A bridge tenant uses the same sequence with a different connection string chosen from the control database after host resolution. The application does not contain a special case that skips set_config. The dedicated database can still run RLS so a bug that points at the wrong database fails closed if the setting does not match any row, and so the code path stays identical.

Technical Explanation

Capacity

Assumptions:

• 20,000 tenants. 50 large (about 10,000 daily learners each). The long tail averages 50 learners. Order of magnitude: large tenants are 50 * 10,000 = 500,000 daily learners and the tail is 20,000 * 50 ≈ 1,000,000 if every small tenant is active, which they are not. Plan 1 million daily active learners across the platform on a busy day, with the 50 large tenants able to burst together on a Monday.
• Sign-in and roster reads dominate. A planning peak of 5,000 requests per second across the data plane, stateless and horizontally scaled.
• One large tenant's import: 1 million enrollment rows in an hour, which is about 300 writes per second sustained, enough to matter on a shared primary if it is unqueued, and easy if it is a fair-queued batch of a few thousand per minute spread so others proceed.
• Usage events: a few per active learner per day, on the order of 10 million events per day. At a few hundred bytes each, this is a few gigabytes per day before rollup. Partition the event table by day. Rollups are tiny.
• Pooled database: one primary, synchronous replica, read replicas for exports. The move-to-bridge trigger is written down: for example, a tenant responsible for more than 25% of primary CPU or I/O for a week, or a contractual residency requirement. The number is a business threshold, not a physics constant. The point is to have one before the incident.

Stateless API pods autoscale on CPU. The primary does not. Protect it with timeouts, fair queues, and the option to move a tenant to bridge. Cache the host map and plan headroom. Do not cache authorization decisions for minutes. A revoked instructor should lose write access on the next request; a few seconds of cached "role = instructor" is the most you should tolerate, and a revocation should delete that key.

APIs we own

Control plane, separate host and credentials:

POST /admin/tenants creates a tenant, a plan, and a hostname. Idempotent on an external customer id.

PUT /admin/tenants/{id}/plan changes quotas. Does not rewrite history.

POST /admin/tenants/{id}/placement with pool, bridge, or silo. Bridge provisions a database and runs migrations, then flips the connection record. The flip is a short cache invalidation of the host map. In-flight requests finish on the old placement; new requests use the new one. A dual-write period is required if you move data rather than only new tenants. Moving a live pool tenant to bridge is a migration project: copy in batches, verify counts, freeze writes briefly, copy the tail, flip. It is not a configuration toggle you promise for tomorrow morning.

Data plane:

All existing product APIs, host-scoped. No tenant_id parameter that grants access. GET /v1/quota shows the tenant admin their meters and headroom. Learners do not need that endpoint.

POST /internal/usage-events is called by product services with a service credential scoped to a tenant and an idempotency key. It is not a public browser API. A browser that could emit its own meter events would under-bill.

Schema

Pooled tables lead with tenant_id. Example for a core enrollment and the meter:

sqlCopy
CREATE TABLE enrollments (
  tenant_id    uuid NOT NULL,
  enrollment_id uuid NOT NULL,
  principal_id uuid NOT NULL,
  course_id    uuid NOT NULL,
  created_at   timestamptz NOT NULL,
  PRIMARY KEY (tenant_id, enrollment_id)
);

CREATE TABLE usage_events (
  tenant_id        uuid NOT NULL,
  idempotency_key  text NOT NULL,
  meter            text NOT NULL,
  quantity         bigint NOT NULL,
  occurred_at      timestamptz NOT NULL,
  source_id        text,
  PRIMARY KEY (tenant_id, idempotency_key)
) PARTITION BY RANGE (occurred_at);

CREATE TABLE usage_rollups (
  tenant_id   uuid NOT NULL,
  meter       text NOT NULL,
  day         date NOT NULL,
  quantity    bigint NOT NULL,
  closed      boolean NOT NULL DEFAULT false,
  PRIMARY KEY (tenant_id, meter, day)
);

Control plane, separate database so a data-plane migration cannot lock tenant records and so data-plane credentials cannot update plans:

sqlCopy
CREATE TABLE tenants (
  tenant_id   uuid PRIMARY KEY,
  hostname    text UNIQUE NOT NULL,
  placement   text NOT NULL,
  plan_id     text NOT NULL,
  region      text NOT NULL
);

Indexes on pooled tables are (tenant_id, ...) for the access path you actually run. A global unique email across all tenants is usually the wrong constraint: the same person may exist in two universities. Uniqueness is (tenant_id, email). If product wants one human identity across tenants, that is a separate platform account linked explicitly, not an accident of a unique index.

Scale, cache, and load balancing

The L7 load balancer terminates TLS for *.cloud.techlearningpro.example and routes to the shared data-plane deployment. It does not parse tenant policy. The application does. Sticky sessions are unnecessary and harmful with pooled bugs: a sticky server that held a wrong SET tenant would stick the victim to the bug. Transaction-local settings make stickiness irrelevant. Prefer no stickiness so any instance can serve any tenant, which is what lets the pool autoscale.

Cache keys are t:{tenant_id}:.... A key without the prefix is a defect the review rejects. Host-map cache TTL is one minute, and rename deletes the key. Quota headroom cache TTL is a few seconds. Session tokens are validated by signature, not stored in a shared map that omits tenant id.

Read replicas serve export jobs and analytics. Sign-in and "did my enrollment commit" read the primary or a replica only if you have measured lag and the product accepts it. Exports of a large tenant run on a replica with a statement timeout so they cannot pin the primary. The fair queue limits how many such exports run at once globally and per tenant.

Bridge placement: the API process holds a small pool of connections per active bridge database, not a pool of 20,000. Idle bridge tenants open on demand and close. A process that connects to every database at startup will fall over at a few thousand tenants. This is the operational reason pool is the default and bridge is earned.

Security

RLS and repository rules as above. Additional points that are easy to miss:

• Search indexes include tenant_id and every query filters it. A search engine that highlights a hit from another tenant is a leak even if PostgreSQL is perfect.
• Logs and traces include tenant_id as a structured field. They do not include other tenants' payloads on error. A stack trace that dumps a request batch can mix tenants; do not log batch bodies.
• Support impersonation mints a short-lived token with tid, an actor, and a reason, written to an audit table in the control plane. It cannot cross hosts.
• Customer-managed keys. For a tenant on envelope encryption, a data key per object or per tenant-period is wrapped by a key in a KMS the tenant can disable, using that KMS's documented wrap and unwrap. Platform operators cannot decrypt when the key is disabled. There is no debug copy. Lesson 58's file bytes are the main consumer. Database columns that are secrets (integration tokens) use the same pattern or a platform KMS if the tenant did not buy CMK. Turning on CMK does not change RLS. Encryption is confidentiality against disk and operator access. RLS is isolation against application queries. They solve different threats.
• Admin APIs are not mounted on the tenant wildcard host. Credential confusion between "I am a tenant admin" and "I am TechLearningPro staff" is a common breach. Separate issuers.
• Webhooks to tenant endpoints sign the body and include the tenant id. A tenant-configured URL is an SSRF risk. Allow only public https endpoints, block link-local and metadata addresses, and resolve DNS at request time with that check. This is the same class of control as any outbound webhook, made sharper because thousands of tenants can configure it.

Failure modes

Trade-offs

Pool versus bridge as default. Pool is cheaper and simpler until a neighbor or a contract forces a move. Bridge isolates I/O and complicates the fleet. Choose pool with an explicit exit ramp, not bridge for the first hundred customers "just in case," or you will spend the year operating databases instead of the product.

RLS versus application-only filters. Application-only is fewer surprises for developers who forget set_config and then see empty lists. That surprise is the safety. RLS-only without application predicates hides missing indexes. Use both, and teach the empty-list failure in onboarding.

Hard quotas versus overage billing. Hard quotas protect the platform and annoy a tenant on graduation day. Overage billing protects the relationship and requires a meter you trust. This design does both: hard quotas on resources that can hurt neighbors (request rate, storage ceiling, import concurrency), overage on contracted commercial meters (active learners) up to a safety ceiling. The safety ceiling still exists so a meter bug cannot generate an unbounded bill or unbounded load.

One region versus tenant-pinned region. Pinning a tenant to a region meets residency and adds a routing rule and a second pooled database. Do it when the contract says so, as a placement attribute, not as a fork. Cross-region active-active for one tenant's rows is a consistency project you do not start by default. A regional pool with asynchronous replicas is the disaster-recovery story for the long tail; RPO is then the replication lag you measured, and you must not call it zero.

Shared search cluster versus per-tenant indexes. Shared is cheaper and needs a mandatory tenant filter. Per-tenant indexes isolate noisy reindex and cost more. Start shared with the filter in the client library you control, not in each call site.

Practical Implementation or Design Walkthrough

Request path. A browser loads https://northwind.cloud.techlearningpro.example. The host map returns tenant 9c1e. The token's tid is 9c1e. The enrollment repository begins a transaction, set_config('app.tenant_id', '9c1e', true), inserts an enrollment whose tenant_id is 9c1e, and appends a usage event with key enroll:{enrollmentId} if the plan meters enrollments. Commit. A bug that inserts tenant_id = eastbridge fails the WITH CHECK policy. A bug that selects without a predicate returns only Northwind rows because of USING.

Import. The admin uploads a roster. The API stores the file via Drive (lesson 58) in that tenant's prefix, enqueues ImportRoster with tenant_id and an idempotency key import:{uploadId}, and returns 202. Workers in the import lane process a page and checkpoint the cursor on the job row. A second message with the same key does not create a second job. The lane's fair scheduler still serves other tenants between pages. Sign-in QPS on the primary stays within SLO. If this tenant's import is still too heavy after the queue exists, placement moves them toward bridge as a project, not as a midnight toggle.

Quota. Before creating an upload session, Drive reads today's storage meter and the plan's hard quota. Over quota, it returns 402 with quota-exceeded and does not start tus. Downloads continue. The tenant admin sees headroom on GET /v1/quota. A soft warning fires at 80% via the notification platform, marketing-class to admins, not a learner passcode channel.

Placement flip sketch. For a new bridge tenant, provision, migrate schema, set placement = bridge and the connection secret reference, invalidate the host cache. For a live move, dual-read verification of row counts per table, a short write freeze, tail copy, flip, and a rollback plan that points the host map back if the sentinel check fails. The sentinel is a known row read with set_config that must return exactly that tenant's marker.

Tests that belong in CI:

• Tenant B's token on tenant A's host is rejected.
• With RLS, a raw SQL client missing the setting reads zero rows.
• A connection returned to the pool does not keep the previous tenant.
• Usage event replay does not change the rollup.
• An import job for A processed on a worker after a job for B cannot see B's rows.
• Cache key helper throws if the tenant id is empty.

Enterprise Real-World Scenarios

Scenario 1: One university's Monday and everyone else's sign-in

A learning platform hosts several hundred institutions in one database. The largest runs a roster sync at 08:00 local that upserts every enrollment in a single transaction and holds row locks. Sign-in p99 for all institutions climbs from 80 ms to multi-second for twenty minutes. The status page looks like a platform outage. It is a neighbor.

The durable fix is the set in this lesson: the sync becomes a paged job on a fair queue with a statement timeout, the upsert transaction covers one page, and indexes lead with tenant_id so the sync does not sequential-scan. They add a per-tenant concurrency cap on the write API. They define the bridge exit: if that university still dominates CPU after the queue lands, they get a database of their own and the same binaries. They do not "temporarily" disable RLS to make the sync faster. The sync was slow because of the transaction shape, not because of the policy. Architecture Performance and Capacity Review uses the before-and-after p99 by tenant, not a single global average that hides the victim tenants.

Scenario 2: A company tenant requires a key they can revoke

A corporate customer will not accept platform-held encryption for assignment files. Placement stays pool for metadata. File bytes use envelope encryption with a customer-managed key, as in lesson 58. When the customer disables the key during a contract dispute, instructors can still sign in and see file names, and content download fails with a clear error. Support cannot "just query the blob." Billing for storage continues for retained ciphertext until erasure runs, which is a contract term, not a surprise. The meter storage_gb_hour counts bytes stored, encrypted or not.

The same customer asks for a dedicated region. That is a placement and residency change: a regional pooled cell or a bridge database in that region, plus a host that routes there. It is not implemented by a column region that nobody reads. The control plane's region selects the cell. A test deploys the tenant into a second cell and proves the first cell's database has no rows for them. Disaster Recovery and Business Continuity then has two stories: platform-wide regional failover for the long tail, and a pinned tenant whose RPO is that cell's replication. Mixing those stories in one slide is how you promise RPO 0 to a tenant whose cell is asynchronous.

Best Practices

• Treat the public catalog as a tenant. Special cases become back doors.
• Resolve tenant from host plus token. Do not authorize from a body field.
• Put tenant_id first in primary keys and in every pooled index you query.
• Enable and force RLS on every pooled table. Set the tenant with set_config(..., true) per transaction.
• Keep a connection-pool test that proves settings do not leak.
• Pass tenant id in job payloads. Do not trust thread-locals across tasks.
• Prefix cache keys and search filters. Review any key that omits them.
• Rate limit and timeout per tenant. Queue heavy work fairly. Autoscale only the stateless tier.
• Separate hard quotas that protect neighbors from commercial overage meters.
• Append usage events with idempotency keys. Invoice from rollups. Correct with new events.
• Migrate pooled schemas expand-contract, backfilling in batches so one tenant cannot lock the table for all.
• Use one codebase across pool, bridge, and silo. Vary placement, not logic.
• Separate control-plane credentials, hosts, and databases from the data plane.
• Audit impersonation and placement changes.
• Define the numeric trigger for moving a tenant to bridge before you need it.

Common Mistakes and Anti-Patterns

Tenant filter only in the UI. The API still returns another campus when called directly. Filters belong in the repository and in RLS. The UI filter is a convenience.

SET on a pooled connection. The next request inherits the tenant. Use transaction-local configuration. Test the pool.

A unique email globally. Two universities cannot both enroll the same address, or worse, you merge their principals. Uniqueness includes tenant_id.

Superuser application role. RLS never runs. The application role is ordinary and cannot bypass.

One queue for all imports and all sign-in side effects. The largest roster becomes everyone's latency. Fair-queue the heavy work. Keep the interactive path on its own pool.

Billing by re-querying live tables. Retries, deletes, and definition changes make the invoice unreproducible. Append events. Roll them up. Close the day.

Forking the service for a silo customer. They miss the security patch. Placement changes the connection and the namespace. The artifact is the same.

Caching a rendered page without the tenant in the key. A CDN or an intermediary serves Eastbridge's roster to Northwind. Private responses are not shared caches. Lesson 57 and lesson 58 made the same mistake in their domains.

Disabling RLS "for the migration." The migration role can be exempt if it is not the application role, and the exemption ends when the migration connection closes. The application role stays forced. A long-lived bypass role used by app servers is a leak waiting for a string change.

Body-supplied tenant id trusted by a worker. Anyone who can enqueue a message can read another tenant. Workers take the tenant from a signed internal token or from a job row written by the API after authorization, not from an untrusted client field alone.

Troubleshooting and Design Review

Module 6 review questions, aligned with Enterprise Architecture Review Checklist:

• What is the default placement, and what metric moves a tenant to bridge?
• Show the RLS policy and the pool-leak test.
• Show a job payload and where tenant context is set on the worker.
• Show a meter event, its idempotency key, and the rollup it affects.
• Show a migration that backfills without an exclusive lock.
• Show cache key format and a negative test for a foreign tenant.

Interview Questions and Answers

Q1. Compare silo, bridge, and pool. What do you run by default?

Pool shares the application and the database, with tenant_id on every row and row-level security behind the application checks. It is the cheapest way to host thousands of small tenants and the widest blast radius for a bad query or a noisy import. Bridge keeps the shared application and gives the tenant their own database, so their locks and disk are theirs and your fleet must migrate many databases. Silo gives them their own deployment as well, which fits residency, dedicated capacity, and awkward change windows, and it costs like a separate install. I default to pool, keep one codebase, and move a tenant to bridge or silo when a written threshold or a contract says so. I do not fork the code for the tenant who shouts loudest.

Q2. How does a request learn its tenant, and what do you refuse to trust?

The hostname maps to a tenant through the control plane. The access token carries tid. If they differ, the request fails. Background work has no host, so the job record written by an authorized API carries the tenant id, and the worker sets database context from that record. I refuse to trust a tenant_id in a browser body or query string as the thing that grants access. I refuse a thread-local left over from the previous job. I refuse a support tool that connects as a superuser and "filters in the WHERE clause when we remember."

Q3. What does row-level security actually buy you?

A policy with USING and WITH CHECK on tenant_id, forced even for the table owner, means a query that forgets the predicate cannot return or insert another tenant's rows as long as app.tenant_id is set locally on the transaction. If it is unset, the comparison fails closed and the caller sees nothing, which is a loud bug. RLS does not replace indexes, so the application still passes tenant_id for the planner. It does not protect Redis, search, or logs. It does not stop a noisy neighbor. It stops the forgotten predicate in SQL, which is the bug that otherwise becomes the incident. The application role must not be superuser and must not have BYPASSRLS.

Q4. Why is SET LOCAL a security control?

A connection pool reuses sessions. A session-level SET app.tenant_id remains when the connection is checked out for the next request, so tenant A's identity authorizes tenant B's work, or the reverse. set_config('app.tenant_id', id, true) lasts for the transaction and disappears at commit or rollback. The regression test checks a connection in, sets A, returns it, checks it out, sets B, and reads. If that test is missing, the setting style will regress the first time someone "simplifies" the repository.

Q5. How do you stop one tenant from slowing the others?

Identify the shared resource. Stateless API CPU can autoscale. A single primary cannot. So interactive traffic has per-tenant rate limits and a concurrency cap, database statements have timeouts, and heavy imports and exports run on a fair queue and on a replica where possible. Storage and notification quotas fail the expensive action rather than the whole platform. If one tenant still dominates the primary for a sustained period, that is the trigger to place them on a bridge database. Autoscaling pods while the primary is locked is how the incident report acquires extra graphs and no fix.

Q6. How do you bill without double-charging retries?

Each billable fact is an append-only usage event with a primary key of (tenant_id, idempotency_key) derived from the source, such as the enrollment id or the file version and the day. Retries insert nothing new. A daily rollup sums events per meter. Invoices read closed rollups. A late event after close writes a correction event and a later adjustment, not an edit of a sent invoice. I do not compute the bill by counting live enrollment rows the night before, because deletes and bugs make that number unreproducible. Meter definitions are part of the contract. Changing active_learner_day from "distinct principal" to "every page view" is a financial incident.

Q7. How do cache, search, and workers stay isolated?

Cache keys start with the tenant id, and a helper refuses to build a key without one. Search requests go through a client that always adds the tenant filter and is tested with two tenants. Workers process messages that contain the tenant id from an authorized enqueue, and they call set_config before any query. A shared in-memory map on the worker is either keyed by tenant or absent. Logs may include the tenant id and must not dump another tenant's record in the same error object. CDN and rendered pages follow the same rule lessons 57 and 58 used: private output is not a shared cache entry.

Q8. How do you migrate schema for 20,000 pooled tenants?

One migration runs against the shared database, so it is everyone's outage if it takes an exclusive lock or rewrites a huge table in one transaction. Expand, backfill in batches that can be tenant-scoped, switch reads, contract. Bridge and silo tenants run that same migration through a fleet job with limited parallelism so you do not open 20,000 connections and do not skip a database. Security patches have a deadline even for silo customers. Feature releases can be tenant-flagged. The artifact is identical; the schedule is not a second codebase.

Q9. When do you offer a customer-managed key, and what does it not solve?

When the contract says the customer must be able to make data unreadable without trusting a platform operator. Envelope encryption wraps a data key with their KMS key using the cloud's documented wrap API. Disabling the key stops decrypt. It does not implement tenant isolation: ciphertext for two tenants can still sit in one bucket prefix scheme if you are careless, so prefixes and RLS remain. It does not by itself meet residency. It breaks deduplication and platform-side malware scanning unless scanning runs where the key is available, which the customer must accept. There is no plaintext side copy for support.

Q10. What is unique about the first-party public catalog?

It is a tenant, with a hostname, a plan, and RLS like the others. Staff are principals in that tenant or they use the control plane, not a bypass role on the data plane. If the catalog code paths "know" they are special and skip the tenant filter, every later feature copies the skip. Shared course templates that other tenants can clone are an explicit publish step that copies rows into the destination tenant. They are not a join across tenant_id from the learner's session.

Q11. A tenant wants a dedicated region and a custom feature next week. What do you agree to?

Region is a placement attribute: their data plane cell is the regional pool or a bridge database, with the host map sending them there, and a test that the other region has no rows. That is a known project. A custom feature next week, if it means a fork, I do not agree to. If it means a tenant-scoped flag on the existing binary, I agree when the flag has a test and a removal date. Silo is available for dedicated compute if they pay for the operation of another deployment of the same artifact. I will not promise a private branch. The capstone in Enterprise System Design Capstone Project can add a feature module on this platform; it should not add a second platform.

Q12. What do you show in a design review to prove isolation?

The host and token check, a repository method that cannot be called without a tenant id, the RLS policy with FORCE, the connection-pool regression test, a worker test that runs tenant A then B on the same thread, a search test, a cache-key test, the meter idempotency key, the fair-queue or timeout control, and the written threshold for bridge placement. I show a migration that does not take an exclusive lock for a backfill. I show that control-plane credentials are distinct. That set is the review. A diagram of boxes labeled "tenant service" is not. System Design Interview Framework can use this list as the close of a multi-tenant question so the interview does not stop at "we'll add a tenant_id column."

Hands-On Exercises

Exercise 1: Place 20,000 tenants

Problem. 20,000 institutions, 50 of them large, one shared product, a public catalog, and invoices based on active learners and storage.

Tasks.

1. Choose a default placement and the trigger to leave it.
2. Specify host and token resolution, including one input you ignore for authorization.
3. Write the RLS policy and the connection-pool test.
4. Define one hard quota and one overage meter, with idempotency keys.
5. Explain how a million-row import avoids owning the primary.
6. Sketch a pooled migration that backfills without an exclusive lock.
7. Say how cache keys and search queries include the tenant.
8. Describe what a customer-managed key does and does not isolate.

Deliverables. A placement note, the policy SQL, the meter key format, and the noisy-neighbor controls.

Exercise 2: Review a "simple" multi-tenant API

The proposal adds tenant_id to tables, filters it in the web controller when the query parameter is present, uses one global unique email, runs roster imports in the sign-in transaction, bills by counting rows on the last day of the month, and connects as a database superuser so migrations are easier. Write the review. Cover leaks, pool reuse, noisy neighbors, billing retries, and the catalog back door.

Rubric.

Lesson Summary

TechLearningPro Cloud isolates customers by resolving the tenant from the host and the token, scoping every row and cache key, and forcing PostgreSQL row-level security as a backstop the application role cannot skip. The default posture is a pool, because most tenants are small and one codebase must carry them. Bridge and silo are the same binaries with a different placement when load, residency, or contract demands it. Noisy neighbors are handled with limits, timeouts, and fair queues, not with hope that the API tier will autoscale a locked primary. Billing is an idempotent event log rolled up into invoices. Migrations assume every pooled tenant feels the lock you take.

Payments, notifications, feeds, files, and dispatch inherit this contract. The Module 6 capstone should add a product on top of it, not a second way to identify a tenant. The interview framework, the security lesson, the capacity review, and the architecture checklist are where you show the tests that prove a missing predicate fails closed.

Further Reading

• Database Sharding and Partitioning, Rate Limiting, Throttling and Load Shedding, API Design, Versioning and Contracts, and Architecture Migration and Modernization.
• Lessons 55–59 in this module, which consume tenant_id on money, messages, feeds, files, and jobs.
• Enterprise Architecture Review Checklist, Architecture Security and Threat Modeling, System Design Interview Framework, and Enterprise System Design Capstone Project.
• Microsoft Azure Architecture Center, tenancy models for multitenant solutions: https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/tenancy-models.
• PostgreSQL row-level security: https://www.postgresql.org/docs/current/ddl-rowsecurity.html.
• PostgreSQL SET and set_config for transaction-local settings: https://www.postgresql.org/docs/current/functions-admin.html.
• Kubernetes ResourceQuota, relevant when a silo tenant has a namespace of their own: https://kubernetes.io/docs/concepts/policy/resource-quotas/.