Design a Global Streaming Platform

Advanced System Design & Software Architecture · Lesson 52 · Real-World System Design Case Studies

Lesson Overview

Course Reel, lesson 51, can package a lesson and authorize playback in one region. TechLearningPro's next product, Atlas, is the same idea with a worldwide learner base: a catalog that must read quickly in Frankfurt and Singapore, a title launch that must not pull tens of millions of segment requests back to one origin, and a legal rule that viewing history for European Union learners stays in the EU. The primer at /learn/system-design/netflix-video-streaming draws a CDN and a transcoder and stops. It does not decide which data are allowed to cross a border, what happens when one CDN provider fails in one country, or how a recommendation row is forbidden from blocking the play button.

Atlas reuses Reel's lifecycle for a single title: immutable HLS generations, tokens, private origin, workers that package once. This lesson designs the global control plane around that lifecycle. Catalog metadata is published as versioned snapshots. Entitlements are evaluated in the learner's region against a replica that is allowed to be seconds behind, except that a suspension must take effect within a token lifetime we choose. Media is encoded once in a home region and replicated to regional origins. Two CDN providers sit in front, with steering that can shift traffic when error rates rise. Recommendations are a read model with a deadline. If they miss the deadline, the home screen still renders an editorial row.

The sequence is lesson 23. Estimates follow lesson 24 and will be larger than Reel's by orders of magnitude on bytes and merely large on catalog QPS. Caching and load balancing show up as snapshot caches and DNS or HTTP steering. Replication is the catalog's consistency story. Sharding is for viewing history by region and user, not for the segment bytes, which do not belong in a database. No proprietary control-plane API of a commercial streaming vendor is assumed. Steering, tokens, and playlists are our contracts plus the public HLS specification already used in lesson 51.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate a global control plane (catalog, entitlement, playback tokens) from a data plane (regional origins and CDNs).
2. Publish catalog snapshots so a region can serve the home screen when the home-region database is unreachable.
3. Estimate global egress, origin fill, and catalog QPS, and show why encoding happens once.
4. Specify regional playback authorization, including suspension freshness and token scope.
5. Design multi-CDN steering and the failure response when one provider degrades in one geography.
6. Keep recommendations off the critical path, with a concrete fallback.
7. Place viewing history so residency constraints survive a support tool in another region.
8. State the trade-off between active-active catalog writes and a single publishing region.

Prerequisites

• Lesson 51, Course Reel. Atlas does not re-specify segment duration, upload resume, or the packager sandbox. Those decisions stand unless a global constraint forces a record.
• Lesson 29, replication, and lesson 31, consistency models. Snapshot publication is a deliberate stale read.
• Lesson 26 and lesson 27, for steering and for cache invalidation of a published catalog version.
• Lesson 24, because the byte math changes the design and the request math does not.
• Lesson 64's disaster-recovery vocabulary is useful and not required. This lesson states the regional failure behavior in its own failure table.
• Comfort with the idea that DNS answers and CDN configurations are part of the system, not a postscript from a network team.

Detailed Core Concepts

Product boundary

Atlas serves a catalog of courses and recorded titles to learners worldwide, authorizes playback in the caller's region, and records progress in that region. Editors still ingest through the home-region pipeline from lesson 51. A separate editorial team publishes "rows" for the home screen: featured titles, continue watching, and a recommendations slot.

In scope:

• A versioned catalog snapshot replicated to each serving region.
• Regional playback authorization using regional entitlement data.
• One encode, many regional origins, at least two CDN providers.
• Home screen assembly with a hard timeout on recommendations.
• Viewing history and resume positions stored in the learner's home region.
• A title-launch pre-positioning step for the first minutes of the expected popular renditions.

Out of scope:

• Live linear channels and sports-scale concurrent live events. The failure mode of live is concurrent encode and a synchronized audience. VOD launch is the problem we size.
• Building our own global anycast network. We contract CDNs and design steering, origin, and token rules. Owning every edge cache is a capital decision, not an architecture diagram.
• A single global user profile that includes EU viewing history in a US primary. Support tools that need a cross-region view receive aggregated or explicitly exported data under a reviewed process, not a standing replica of the history tables.
• Personalized encoding per device. The ladder from lesson 51, possibly extended by one 4K rung for a subset of titles, is encoded once.
• Replacing the catalog team's source of truth with the search index or the recommendation model.

Functional behavior

A learner opens the home screen in a region chosen by a stable mapping: the account's residency region, not the VPN of the hour, for data placement. The read of the catalog uses the latest snapshot that region has fully downloaded. The response includes editorial rows and, if the ranker answers in time, a personalized row. Continue-watching is regional history, not the snapshot.

Play requests hit the regional authorization service. It loads the title pointer (which generation is current) from the snapshot, checks entitlement against the regional entitlement store, and returns a CDN URL and a token. The token names the provider, the prefix, and the expiry. The player uses HLS as in lesson 51.

Editors publish a catalog version by freezing a set of title pointers, artwork references, and synopses. Regions download the snapshot, verify a checksum, and atomically switch the version they serve. A partial download is not served. Rollback is a switch to the previous version the region still holds.

Suspension of an account sets a flag in the entitlement store and replicates. Tokens already issued live until expiry. New authorizations fail. The expiry we pick is the suspension window. That is the same honesty as Reel's unpublish window, applied to a person rather than a title.

Quality scenarios

Home screen. A learner in a served region receives a catalog response in p95 under 200 ms at the regional edge, from snapshot data, with recommendations omitted if the ranker exceeds 80 ms. The screen is allowed to be missing the personalized row. It is not allowed to wait indefinitely or to fail the whole page.

Play start. Regional authorization p95 under 150 ms when entitlement data is local. Master playlist from a warm edge p95 under 400 ms, the same player-visible budget as lesson 51, now measured in each served region, not only at home.

Snapshot freshness. A title published in a catalog version is visible in every healthy region within 5 minutes of the version being marked released, p95. This is not a promise that a learner who already has a home screen open will repaint without a refresh. Clients refresh on a timer of a few minutes and on play.

Suspension. After the entitlement flag commits in the account's region, new playback authorizations in that region fail within 5 seconds, and in other regions within the replication lag bound we set at 60 seconds, plus any token already issued, which expires in 15 minutes. Cross-region suspension is therefore "about a minute to stop new tokens, up to 15 minutes to stop existing ones." Anyone who wants instant global kill of bytes in flight needs a denylist pushed to CDNs, which is an additional control with its own failure mode. v1 writes the window and operates the denylist only for incidents.

Region loss. If a serving region's snapshot store and authorization service are down, learners are steered to another region only if their residency rules allow that region to see their entitlement. EU learners are not silently sent to a US authorization database. They receive a degraded "service unavailable" rather than a residency breach. Catalog browsing from a snapshot copy in a second EU zone is the designed degradation. Playback that requires entitlement may pause. Bytes already cached at a CDN can continue for holders of valid tokens.

Publication, not distributed writes

Two ways exist to run a global catalog.

Active-active writes. Any region can edit a title. Conflicts appear: two synopses, two "current generation" pointers. You need a conflict policy, and the learner can see oscillations.

Single writer, published snapshots. Editors write in the home region. A version is an immutable object: a manifest of title ids, generation ids, text, and artwork checksums. Regions consume versions. The home region can be down and regions still serve the last good version. Edits wait. For a course catalog, waiting for edits during a home-region incident is acceptable. Waiting for playback is not, and playback uses the last snapshot plus regional entitlements.

Atlas chooses publication. The catalog primary is not on the play path. Lesson 29's replica lag still exists inside a region for entitlements, and we bound it. We do not pretend a worldwide quorum of catalog writes.

Artwork and text in the snapshot are small. The snapshot for 100,000 titles, a few kilobytes of metadata each, is on the order of a gigabyte or less of JSON or a columnar file, checksummed, stored as an object, pulled by each region. That is a file transfer, not a query fan-out. Services in the region memory-map or index the snapshot and serve reads locally. Search may build its own index from the same version so search and home screen agree on what "released" means. If search lags, the home screen is still the snapshot. Play authorization uses the snapshot's generation pointer, not the search index. A search hit that is not in the snapshot does not play. This avoids the class of bug where the index promotes a title the publisher pulled.

Encode once, replicate bytes

Encoding a 4K ladder for a large library is expensive in CPU and in human QA. Doing it in every region multiplies cost and risks divergent segments. Atlas encodes in the home pipeline, writes an immutable generation, and replicates objects to regional buckets. Replication is a copy of bytes that content-address or generation-address already made immutable, so a re-copy is safe and a partial copy is not published. A region's origin serves a generation only after the replicator marks that generation complete. The catalog snapshot must not point at a generation a region cannot serve. The publish tool checks replica completeness for the regions in the title's availability set before it releases the version. A title not cleared for a region does not appear in that region's snapshot.

CDN providers pull from the nearest healthy origin. An origin shield per region collapses edge misses. Pre-positioning fetches the first minute of the mid and low rungs through each CDN after release, using the same URLs a player would use, so the fill is cached. Pre-positioning is best effort and is mandatory in the launch checklist for a featured title because the math below shows a cold cache is an origin incident.

Multi-CDN steering

One provider will have a bad hour in one geography. Atlas advertises two provider hostnames and chooses at playback time, in the authorization response, based on a health table: recent error rate and throughput by provider and country, measured from player beacons and from synthetic probes. The token is minted for the chosen hostname. A player that sees repeated segment failures may call authorize again and receive the other provider. That retry is part of the client contract.

Health data lags. Steering can be wrong for a minute. The alternative, a single CDN, is wrong for the whole incident. We accept oscillation risk by adding hysteresis: switch a country away from a provider when error rate exceeds a threshold for several minutes, and switch back only after a longer calm period. Document the thresholds in configuration, not in folklore.

DNS-only steering is coarser and caches in resolvers. Authorization-time choice is per play and respects token scope. We use authorization-time choice as the primary lever and keep DNS as a coarse default for artwork and other non-token assets.

Recommendations as a boundary

The ranker reads viewing history in-region and returns a list of title ids. The home-screen service waits at most 80 ms, then fills the row from the editorial default. The ranker is not called on the play path at all. A bad model can bore people. It cannot stop a lesson from starting. Timeouts fail open to editorial content. A ranker error does the same. The snapshot still filters the list: the ranker may not introduce a title the version does not contain or the region may not show.

This is a lesson 2 quality decision written as a timeout. Teams that "just call the model synchronously" will discover that model latency is the home-screen latency. The 80 ms budget makes the dependency negotiable.

Residency

Each account has a residency region chosen at registration and changed only through a reviewed workflow. Viewing history, resume position, and precise entitlement state live there. Catalog snapshots are not personal data; they replicate globally. Tokens must not embed the history. Support staff in another region see account status and non-content metadata. They do not get a replica of the EU history table to "make debugging easier." A break-glass export is an audited, time-limited object, not a pipeline.

If the residency region's history store is down, continue-watching disappears and playback of already entitled titles can continue if entitlement has a local replica that is still inside its lag bound. We do not fail over history writes to another region. Resume positions queue or wait. Losing a resume point for a few minutes is a better failure than writing EU viewing data into a US store during an incident and calling it temporary.

What the global primer still skips

The consumer primer stops at "CDN plus transcode." Atlas has to publish versions, refuse cross-border history failover, steer between CDNs with hysteresis, and degrade the home screen when the ranker is slow. Those are the design. The ladder and the segment format are already decided.

Architecture and Design Diagrams

mermaidCopy
C4Context
    title Atlas — system context
    Person(learner, "Learner", "Browses and plays in a residency region")
    Person(editor, "Editor", "Publishes catalog versions from the home region")
    System(atlas, "Atlas", "Snapshots, regional authorization, CDN steering")
    System_Ext(reel, "Course Reel pipeline", "Encodes a generation once")
    System_Ext(cdn, "CDN providers", "Two providers, many edges")
    System_Ext(ranker, "Recommendation ranker", "Optional personalized row")
    Rel(learner, atlas, "Home and authorize", "HTTPS")
    Rel(editor, atlas, "Releases a catalog version", "HTTPS")
    Rel(atlas, reel, "Reads completed generations", "HTTPS")
    Rel(learner, cdn, "Fetches HLS", "HTTPS")
    Rel(atlas, ranker, "Asks with a deadline", "HTTPS")

mermaidCopy
flowchart TB
    subgraph home [Home region]
        edit[Publish tool]
        primary[(Catalog primary)]
        encode[Reel encode pipeline]
        originH[(Home origin bucket)]
    end
    subgraph eu [EU serving region]
        snapE[Snapshot reader]
        authE[Authorization]
        histE[(Viewing history)]
        originE[(EU origin)]
    end
    subgraph apac [Second serving region]
        snapA[Snapshot reader]
        authA[Authorization]
        originA[(Regional origin)]
    end
    cdn1[CDN provider A]
    cdn2[CDN provider B]

    edit --> primary
    encode --> originH
    originH --> originE
    originH --> originA
    primary --> snapE
    primary --> snapA
    snapE --> authE
    histE --> authE
    authE --> cdn1
    authE --> cdn2
    cdn1 --> originE
    cdn2 --> originE
    cdn1 --> originA
    cdn2 --> originA

mermaidCopy
sequenceDiagram
    participant Ed as Editor
    participant P as Publish tool
    participant O as Origins
    participant R as Regional reader

    Ed->>P: release version N
    P->>O: confirm generation replicated
    P->>P: write snapshot object and checksum
    R->>P: download snapshot N
    R->>R: verify checksum
    R->>R: switch served version

mermaidCopy
sequenceDiagram
    participant L as Learner
    participant H as Home screen
    participant Rank as Ranker
    participant A as Authorization
    participant C as CDN

    L->>H: GET home
    par catalog
        H->>H: read local snapshot
    and rank
        H->>Rank: recommend, 80 ms budget
    end
    alt ranker in time
        H-->>L: editorial rows plus personalized row
    else timeout
        H-->>L: editorial rows only
    end
    L->>A: authorize title
    A->>A: entitlement in region
    A-->>L: URL for chosen CDN, token
    L->>C: HLS playback

Technical Explanation

Snapshot readers

Each region runs a reader that polls for a new version number, downloads the object, checks the checksum, builds an in-memory index by title id, and flips an atomic reference. Requests in flight keep the old reference until they finish. There is no partial version visible. Memory for a few hundred megabytes of metadata is cheap next to egress. If the download fails, the region keeps serving the previous version and pages a human. Serving a half-applied file is worse than being five minutes stale.

The home screen and authorization both read this index. They do not call the home primary. A bug that "falls back to the primary for freshness" will turn a home-region database blip into a worldwide outage and will also violate the idea that play does not depend on the writer. Freshness is the next snapshot. The 5-minute scenario is the SLO for the poll-and-flip pipeline, monitored by version-age per region.

Entitlement and tokens

Entitlement is "this account may play this title now," derived from enrollment, subscription, and suspension. The regional store is a replica of the account's residency region if those differ, or the local primary if the learner is home. Atlas encourages residency region and serving region to be the same. Travel is the exception: a learner physically elsewhere still authorizes against residency data through a read replica in a region allowed to hold it, or the client talks to the residency region's authorization endpoint. We do not copy the entitlement row into a random edge because the user took a flight. Latency for those users can be worse. The scenario for play start is written for learners served in-region. Travelers are a known slower path.

Tokens last 15 minutes, bind provider hostname, object prefix, generation, and an opaque subject. Refresh is a new authorize call. The player does this before expiry during long lessons, as lesson 51 required. Suspension between refreshes waits for expiry unless incident response pushes a denylist. The denylist is a small set of subject or token ids replicated to the CDN configuration on an incident timer, not on every suspend. Most suspends are covered by "no new token."

Steering and load

Authorization is stateless behind a regional load balancer. Any instance can sign a token if it has the current key and the snapshot reference. Keys rotate with overlap, as in lesson 51. Health scores for CDNs are computed by a small job and written to a table the authorizer caches for a few seconds. A thundering herd of authorizations at title launch is real but still request-shaped: even a million starts spread over ten minutes is about 1,700 requests per second in a region. That is a modest horizontally scaled service. The bytes those starts trigger are not modest. Scale the authorizer by CPU of signature and snapshot lookup. Scale the data plane by pre-position and by having two CDNs that already hold the prefix.

Player beacons: the client reports startup time, rebuffer count, and failed segment hosts. Beacons go to a regional collector, sampled if volume requires, and feed steering after aggregation. They are not on the play path. A beacon outage must not block playback. Fail open by ignoring beacons and using synthetic probes only.

Replication of objects

The replicator copies generation prefixes and writes a manifest of object counts and checksums. The publish tool queries that manifest. A late object delays the version for that title or drops the title from the version, which is an editorial choice: slip the title, do not slip the whole catalog. Prefer per-title readiness so one stuck replicate does not block an unrelated course.

Delete and unpublish follow lesson 51's token and purge rules on every CDN and every origin. A region that still has bytes after unpublish is fine if no token will be signed and purge has been requested. Garbage collection of old generations waits until every region's snapshot has moved on and token expiry has passed, then deletes. A sweeper that deletes "anything not in the latest snapshot" without waiting for token expiry will break learners mid-video. The sweeper's clock includes the 15-minute token plus a purge buffer.

Failure of a CDN or an origin

If provider A errors in one country, steering shifts new authorizations to provider B after hysteresis. Existing tokens for A continue until expiry or until the player retries authorize on segment failure. Both providers must be able to pull from the regional origin. If the regional origin is down and the home origin is allowed to serve that region's content, the CDN configuration can fail over origin. For EU titles whose bytes are permitted to exist in EU origins only, there is no US origin failover. The second EU zone's origin is the failover. That costs a second copy inside the residency boundary. The estimate should include it. Saving the copy and failing over across the boundary is a legal incident, not a clever reliability trick.

Caching summary

Lesson 27's rule holds: the snapshot staleness is the publication SLO, and the segment cache is safe because the bytes do not change.

Security

Regional services authenticate users with the company IdP. Tokens are signed by Atlas, verified by the CDN. Artwork URLs may be public if the artwork is not sensitive; lesson video is not. Separate hosts keep a public cache from being confused with a token-gated one.

A snapshot is integrity-protected by checksum and by fetching it over TLS from the publisher. A region will not flip to a version whose checksum does not match the signed release record. Otherwise a partial or substituted file becomes the catalog.

Entitlement checks fail closed. Ranker output is filtered by the snapshot and by geo availability. The ranker is untrusted input. It proposes ids; the home service disposes.

Support access across regions uses explicit tools and audit, not database credentials to the history primary. Break-glass is time boxed.

Rate limits on authorize prevent a leaked account from minting infinite tokens. The CDN still enforces token expiry and prefix. Both layers exist because either one will be misconfigured eventually.

Trade-offs

Publication latency versus write availability. Editors cannot update the catalog if the home region is down. Learners can still play the last version. We take that trade.

Two CDNs versus one. Cost and operational complexity go up. A single-provider incident stops being a total outage. Hysteresis is the tax we pay to avoid flapping.

Encode once versus regional encode. We save CPU and avoid divergent ladders. We pay replication delay before a title can be released in a far region. The 5-minute snapshot SLO assumes bytes are already there. A new title's critical path includes replication, which can be longer than five minutes for a large generation. The publish tool waits. Do not encode the "5 minutes" slogan into a promise about a 50 GB 4K generation crossing an ocean. State both clocks.

Fail open recommendations versus fail closed. Fail open protects the home screen and can hide a ranker outage in product metrics. We alarm on fallback rate so "open" does not become "silently dead."

Traveler latency versus copying entitlement everywhere. We keep entitlement in residency. Travelers pay latency. The alternative fails the residency scenario.

Practical Implementation or Design Walkthrough

Assumptions

These are planning assumptions for Atlas, not measurements of Course Reel. Label them when you reuse the packet.

Arithmetic

Egress:

[ 5 \times 10^6 \times 45 \times 60 \times \frac{3 \times 10^6}{8} \text{ bytes/day} ]

Forty-five minutes is 2,700 seconds. Times 3 Mbps is 8,100 megabits, about 1.0 GB per learner per day. Times 5 million learners:

[ 5 \times 10^6 \text{ GB/day} = 5 \text{ PB/day} ]

Order of magnitude: about 5 petabytes of egress per day. A 97 percent hit ratio leaves about 3 percent at origins:

[ 0.03 \times 5 \text{ PB} \approx 150 \text{ TB/day} ]

spread across regions and titles. A single new title watched by 10 percent of DAU in one evening is (0.5 \times 10^6) learners times about 1 GB if they watched the full 45 minutes, or less if they sample. Even 10 minutes each is about (0.5 \times 10^6 \times 225) MB ≈ 110 TB of one title, much of it in a few hours, and the first minute of that is a cache-fill storm. Origin shielding and pre-position are sized for the first segments, not for serving the petabyte. If your design's origin is a single bucket in one region with no shield, 150 TB/day average and launch spikes will exhaust it. Regional origins exist because of that number and because of residency, not because diagrams look better with more cylinders.

Catalog storage: (100{,}000 \times 4) GB = 400 TB per full replica. Three zones are about 1.2 PB before overhead and before two CDN caches, which are not our disks but are our bill. Encoding once does not mean storing once.

Snapshot size: (100{,}000 \times 2) KB = 200 MB. Trivial to pull every region. The publication design is not constrained by snapshot bytes. It is constrained by correctness of the flip and by generation replication of the large objects.

Home-screen QPS:

[ \frac{5 \times 10^6 \times 20}{86{,}400} \approx 1{,}160 \text{ requests/s average} ]

Peak factor 5 is about 6,000 requests per second worldwide, divided by regions. Snapshot reads in memory handle this. The ranker, if called for every home fetch, must shed load via the 80 ms timeout or it will become the outage. Assume half of home fetches ask the ranker. If the ranker cannot keep 80 ms at that rate, more learners get the editorial row, which is the designed degradation. Measure fallback rate.

Authorization QPS:

[ \frac{5 \times 10^6 \times 3}{86{,}400} \approx 174 \text{ requests/s average} ]

Peak around 1,000 requests per second worldwide, higher in a launch region. Horizontal API scale is straightforward. Do not put this QPS on the catalog primary.

Encode cost is why "package in every region" fails the design. A 4 GB packaged title that took worker-hours to produce, times 100,000 titles, is a large but bounded bill. Times three regions of encode would triple it and create drift. Replication of 400 TB is a network project with a schedule. It is still cheaper and safer than three encoders that disagree.

API

GET /v1/home

• Regional. Auth required. Response: catalogVersion, editorial rows, optional personalized row or personalized: null with fallback: true. Cache-Control: private, max-age=60.

POST /v1/playback/authorizations

• Body: titleId. Response: cdnHost, playlist URL, token, expiresAt, catalogVersion, generation.
• 403 not entitled, suspended, or not in this region's snapshot.
• 409 generation not replicated here.
• 503 entitlement store unavailable. No token.

GET /v1/catalog/versions/current

• Operations. Returns the version id each region reports. Used by the freshness SLO.

POST /v1/catalog/releases

• Editors. Body: version manifest checksum. The tool, not a human, calls this after replication checks. 409 if a referenced generation is incomplete in a required zone.

POST /v1/beacons/playback

• Sampled player reports. 202. Never required for play. No auth secrets in the body beyond a session id.

History writes:

PUT /v1/history/{titleId} with position seconds, regional, idempotent on (user, title, position bucket) so a retry does not create duplicate events. Precise event design can be a small row updated in place for "current position" plus an append-only log sampled for the ranker. The current position is the user-visible fact and is the one we must get right. The ranker can lag.

Schema

Catalog primary, home region only:

sqlCopy
CREATE TABLE catalog_release (
    version_id    bigint PRIMARY KEY,
    checksum      char(64) NOT NULL,
    released_at   timestamptz,
    status        varchar(16) NOT NULL
);

CREATE TABLE title_pointer (
    version_id   bigint NOT NULL,
    title_id     uuid NOT NULL,
    generation   bigint NOT NULL,
    available_in varchar(16) NOT NULL,
    PRIMARY KEY (version_id, title_id, available_in)
);

Regional entitlement and history, not replicated across residency boundaries:

sqlCopy
CREATE TABLE entitlement (
    account_id  uuid NOT NULL,
    title_id    uuid NOT NULL,
    state       varchar(16) NOT NULL,
    updated_at  timestamptz NOT NULL,
    PRIMARY KEY (account_id, title_id)
);

CREATE TABLE resume_position (
    account_id  uuid NOT NULL,
    title_id    uuid NOT NULL,
    position_s  integer NOT NULL,
    updated_at  timestamptz NOT NULL,
    PRIMARY KEY (account_id, title_id)
);

available_in on the pointer stops a snapshot builder from shipping a title into a region whose origin does not have the bytes. The regional reader never consults this table directly; it consults the snapshot the builder produced from it. One path.

CDN health is operational data, short retention:

sqlCopy
CREATE TABLE cdn_health (
    provider    varchar(32) NOT NULL,
    country     char(2) NOT NULL,
    error_rate  real NOT NULL,
    updated_at  timestamptz NOT NULL,
    PRIMARY KEY (provider, country)
);

Spring notes

Regional authorization is a Spring service with the snapshot held in an AtomicReference. The controller does not call the home JDBC URL. A test can swap the reference. Token signing is a port so unit tests assert prefix and expiry without a CDN.

javaCopy
public Authz authorize(Account account, UUID titleId) {
    CatalogSnapshot snap = published.get();
    TitlePointer title = snap.find(titleId);
    if (title == null) {
        throw new NotInCatalogException();
    }
    Entitlement ent = entitlements.get(account.id(), titleId);
    if (ent == null) {
        throw new EntitlementUnavailableException();
    }
    if (ent.suspended() || ent.denied()) {
        throw new ForbiddenException();
    }
    Provider provider = steering.choose(account.country());
    return tokens.issue(title, provider, Duration.ofMinutes(15));
}

The home assembler uses a CompletableFuture with an 80 ms timeout for the ranker and always has the editorial rows in hand first, so a timeout still returns 200. Do not start the editorial query after the ranker. That inverts the dependency.

The publish tool is a batch job, not a request thread: verify replica manifests, write the snapshot object, insert catalog_release, then flip released_at. Regions poll released_at. A release without released_at is invisible. That two-phase flag prevents a reader from seeing a row before the object exists.

Tests

• A region with version N ignores a corrupt N+1 download and keeps serving N.
• Authorize uses the snapshot generation, not a stale search document that names an older one.
• Ranker sleeping past 80 ms yields fallback: true and 200.
• Entitlement timeout yields 503 and no token.
• Steering hysteresis: a one-second error spike does not flip providers; a sustained breach does.
• A title not marked complete in the EU replica cannot be released into the EU snapshot.
• Resume writes stay in the residency database; a test double for the other region never receives the row.
• Player retry calls authorize again and may receive the second host.

Records

ADR-052-1. Catalog changes are published versions from one writer. Regions serve the last complete snapshot. Rejected: active-active edits; play path reads of the home primary. Reopen if editors must change metadata while the home region is down and that requirement outranks simplicity.

ADR-052-2. Two CDNs, chosen at authorization, with hysteresis. Recommendations fail open in 80 ms. History does not fail over across residency boundaries. Rejected: one CDN; ranker on the play path; "temporary" history writes to another continent during an outage.

Enterprise Real-World Scenarios

The featured premiere

A new course is featured in every region at 18:00 local time, which means the storm rolls across time zones rather than hitting once. In the first region, a million learners are eligible and a large fraction open the app. Home-screen QPS in that region jumps. The snapshot holds. The ranker times out for many of them, fallback rate alarms, and the editorial row still shows the premiere. That is success, not an incident, as long as play works.

Play starts mint tokens for provider A. Pre-position filled the first minute on A and B. Segment error rate stays low. The failure we rehearse is pre-position being skipped. Origin bandwidth in that region pegs, start-up time exceeds the scenario, and steering may even shift to B, which is also cold, doubling origin pain. The launch checklist gates the feature flag on a pre-position report: object hit counts for the first segments on both providers. No report, no feature on the home row. Editorial enthusiasm does not bypass the gate. The bytes will not negotiate.

EU history and a US support incident

A learner in Germany cannot resume. A US support engineer opens the admin tool and expects the same screen they have for US learners. The tool shows entitlement and "history unavailable in this console; request an export." The engineer is annoyed and asks for a replica. The design refuses. The on-call in the EU region reads resume_position there. If the EU history store is the thing that is down, resume waits, and playback of the title still authorizes from entitlement if that store is up. The incident channel will pressure someone to point the EU service at the US database "for an hour." The failure table and ADR-052-2 exist so that hour does not happen under stress. An export of one account, audited, is the path when a specific case needs a human outside the region. A pipeline is not.

A second variant: catalog primary down during a typo fix. Regions keep serving the last version. The typo remains for the duration. Communications say "playback is up, metadata edits are paused." That sentence is only true if nobody added a hidden primary read on the play path. The synthetic probe authorizes and fetches a segment in each region without using the primary. If the probe needs the primary, the probe is testing the wrong design.

Best Practices

• Publish immutable catalog versions. Do not read the writer on the play path.
• Encode once, replicate bytes, and block release on replica completeness per region.
• Choose CDNs per authorization and switch with hysteresis. Let the player ask again after segment failures.
• Put a deadline on recommendations and ship the editorial page anyway. Alarm on the fallback rate.
• Store viewing history in the residency region and do not fail it over across the boundary.
• Size origin for launch misses, not for the average hit ratio.
• Keep tokens short, scoped to provider and prefix, and refresh them from the player.
• Separate public artwork caches from token-gated media hosts.
• Include token lifetime in the garbage-collection clock for old generations.
• Probe play without the catalog primary so a fallback cannot sneak into production.

Common Mistakes and Anti-Patterns

Active-active catalog edits with last-write-wins. Synopses and generation pointers will flap. Learners will not know which cut is current. Publication removes the race.

Encoding in every region "for locality." You pay triple CPU and you will eventually serve different bits under the same title id. Replicate immutable objects instead.

One CDN because the contract was easier. The first regional incident becomes a title launch failure. Two providers is the minimum if this product is the company's storefront.

Ranker in the same request as play, with no timeout. Model latency becomes an outage of education. The ranker does not belong on authorize at all.

Failover of EU history to the US primary. It will be described as temporary and will become the backup plan. Refuse it in the record.

Trusting a 97 percent hit ratio on premiere night. The cache is empty for that prefix. Pre-position or suffer the origin.

Deleting the previous generation as soon as the snapshot flips. Tokens still point at it. Wait out expiry and purge.

Falling back to the catalog primary when the snapshot is "a bit old." You have re-coupled the world to one database and undone the publication design.

Putting the user id and email in the segment query string. Tokens get copied into logs and screenshots. Use an opaque subject and a prefix signature.

Troubleshooting and Design Review

Symptom: One region shows yesterday's featured row; others show today's.

Likely cause: snapshot flip failed checksum, poller stopped, or the region is serving a cached home response while the probe looks at the reader.

Investigation: version id on the reader, checksum errors, client catalogVersion, poller lag.

Corrective action: fix the download path. Do not point the region at the primary for reads. Rollback is serving the last good version on purpose; being a day behind is a stuck poller, not a rollback.

Symptom: Start-up time explodes in one country and origin bytes spike.

Likely cause: missed pre-position, shield bypass, or both CDNs missing and retrying origin for every segment.

Investigation: hit ratio by title and provider, shield logs, launch checklist artifact.

Corrective action: pre-position, restore shield, steer to the provider that actually filled. Scale origin only as a bridge.

Symptom: Suspended learners still play for half an hour.

Likely cause: token lifetime, a client that does not refresh and already has a token, replication lag of the suspend flag, or a region still using an entitlement cache longer than the scenario.

Investigation: token exp, entitlement updated_at per region, cache TTL.

Corrective action: align cache with the 60-second cross-region bound. For an active incident, use the denylist. Do not claim the window is zero.

Symptom: Personalized row disappears for everyone and nobody is paged.

Likely cause: ranker over the deadline, fallback working, alarm missing or muted.

Investigation: fallback rate, ranker latency, home p95 which still looks fine.

Corrective action: page on fallback rate, not only on home errors. The design hides the outage from learners on purpose, so the metric must not hide it from you.

Symptom: A title plays in the US and returns 409 in the EU.

Likely cause: generation not replicated, or available_in omitted the EU and the client is hitting a stale search result that still lists it. If search and snapshot disagree, authorization is right to refuse.

Investigation: replica manifest, snapshot contents, search document version.

Corrective action: finish replication and release a version, or remove the search document. Do not special-case authorize to "try the home origin" for EU accounts.

Design-review questions: What does a region serve if the home primary is down? How long may a snapshot lag? What is the suspension window including tokens? Where is viewing history written, and where is it forbidden? What happens when the ranker takes a second? How do you know a premiere will not melt the origin? What prevents an old generation from being deleted under an active token?

Interview Questions and Answers

Q1. How is a global catalog kept consistent enough to play the right cut?

Editors write in one home region. A release is an immutable snapshot of title pointers and text, checksummed, downloaded by each region, and flipped atomically. Play and the home screen read the snapshot, not the primary. Lag is bounded by the publication SLO, about five minutes for metadata once bytes are ready, and a new title also waits for object replication, which can be longer. I reject active-active edits because two generation pointers for one title is a worse product bug than a short publish delay.

Q2. Why encode once?

A full library at several gigabytes packaged per title is already hundreds of terabytes. Encoding that in every region multiplies CPU and risks regions serving different media for the same generation id. Immutable copy to regional origins preserves one bit-stream and satisfies residency by where the copy lives. The publish tool refuses to list a title in a region whose replica is incomplete.

Q3. What is your egress math, and what does it change?

Five million daily actives, 45 minutes, about 3 Mbps, is on the order of a gigabyte per learner per day, about 5 PB/day. A 97 percent CDN hit ratio still leaves on the order of 150 TB/day at origins, and a premiere is a cold prefix that the average hit ratio does not describe. So the design is regional origins, shields, two CDNs, and a pre-position gate. Authorization QPS is only hundreds to low thousands at peak and is not the driver. Anyone who shards the catalog database and proxies video through it has inverted the bottleneck.

Q4. How do you use two CDNs without flapping?

Authorization picks a provider from a health table keyed by country and signs the token for that host. We switch away only after error rates stay bad for several minutes, and we switch back after a longer recovery. The player that sees segment failures calls authorize again and can land on the other provider. DNS remains a coarse default for non-token assets. Health beacons are off the play path.

Q5. What happens when the recommendation service is slow or wrong?

The home screen waits at most 80 ms and then returns editorial rows with a fallback flag. Play authorization never calls the ranker. Suggestions are filtered to titles present in the snapshot and allowed in the region. I alarm on fallback rate because the user-visible page still looks healthy. A wrong but fast ranker is a product issue. A slow ranker must not be an outage.

Q6. How do you suspend a learner globally?

The entitlement flag replicates to regions that are allowed to hold it. New authorizations fail within the replication bound, targeted around a minute across regions and seconds locally. Tokens already issued last up to 15 minutes. An incident denylist can push a kill to the CDN for the rare case where that window is too long. I do not promise that buffered video on a device vanishes. I also do not replicate EU history to the US to make the suspend "easier to see."

Q7. A region's origin fails. What do you fail over to?

To another origin inside the same residency boundary, fronted by the CDNs. I do not point an EU learner's playback at a US bucket to save the incident. If no in-boundary origin is healthy, playback fails and browsing can continue from the snapshot. Tokens and prefixes stay the same shape so the CDN origin configuration is the failover lever, not a second application path that proxies bytes.

Q8. Why are travelers a special case?

Data placement follows the account residency, not the IP of the cafe. A traveler authorizes against residency data, possibly with worse latency, or the client uses the residency endpoint. Copying entitlement and history to whatever edge they touched would spray personal data worldwide. The play-start SLO is written for in-region learners so we do not pretend the traveler path is equally fast.

Q9. How do snapshots interact with search?

Search indexes a catalog version and can lag. Authorization trusts the snapshot only. A search hit that is absent from the snapshot does not play. Home rows come from the snapshot plus optional ranker ids that are filtered the same way. One version id in the client response makes mismatches visible in logs.

Q10. Where do caching and load balancing matter?

The useful cache is the CDN on immutable segments, the in-memory snapshot, and a few seconds of CDN health. Authorization responses are no-store. Regional authorizers sit behind a stateless load balancer and scale on CPU. The balancer does not pin play to a single node. Sticky sessions would not help a stateless signer and would hurt deploys.

Q11. What would you shard?

Viewing history and resume rows shard by account id inside the residency region if one database cannot hold the write rate. Catalog metadata at 200 MB per version does not need a shard; it needs a file. Segment bytes are objects, not rows. I would not shard the catalog primary to look ready for scale. Lesson 30 applies to the history store when measurements say so.

Q12. How is this different from the single-region video lesson?

Lesson 51 designs upload, packaging, generations, and one CDN for course video. Atlas assumes that pipeline, then adds versioned publication, regional origins, residency of history, multi-CDN steering, launch pre-positioning, and a ranker that is allowed to be skipped. I reuse the 15-minute scoped token and the immutable generation. I do not reopen segment duration here unless global request cost forces a new record.

Hands-On Exercises

Exercise: Regional failure packet

Problem statement. The EU authorization database is unavailable for 40 minutes. CDN edges and both EU origins are healthy. The home-region catalog primary is healthy. EU learners are in the middle of an evening. US learners are unaffected. Write the degradation behavior and the controls that prevent a well-meaning failover from moving EU personal data.

Tasks.

1. List what still works: snapshot browse, resume display, new play, in-progress play with a valid token, ranker.
2. State what you will not do with the US entitlement store and the US history store.
3. Estimate how many EU learners hold a token that still works, using an assumption you label about evening concurrency and the 15-minute token.
4. Specify client-visible status codes for home and for authorize during the outage.
5. Draw the dependency path and mark which arrows are allowed to be used.
6. Define the probe that must fail and the probe that must stay green.
7. Write the incident communication sentence for learners and the one for support.
8. Record an ADR addendum: conditions to restore service, and the rejected shortcut.

Expected deliverables. Degradation list, arithmetic, diagram, probe list, ADR note.

Evaluation rubric.

A packet that restores play by pointing the EU authorizer at the US primary is a failed review, even if the diagram is tidy.

Lesson Summary

Atlas is a global control plane around a single encode pipeline. Catalog readers serve versioned snapshots and do not call the home primary on the play path. Entitlements and viewing history stay inside the learner's residency region. Media bytes replicate to regional origins and reach learners through two CDNs chosen at authorization time, with hysteresis and a player retry. Recommendations have 80 milliseconds and then lose to the editorial row.

The capacity story is about five petabytes of egress a day at the stated assumptions, residual origin traffic that still needs shields, and launch spikes that a 97 percent average hit ratio does not cover. Catalog QPS is real and solvable with an in-memory snapshot. The failures that matter are a stuck snapshot, a cold premiere, a CDN degradation, a suspend window people must understand, and a residency shortcut someone will propose during an incident.

Lesson 51 built one title's lifecycle. This lesson made that lifecycle survivable worldwide without pretending the catalog is a single database query. Lesson 53 changes the problem again: the scarce resource is a nearby driver and a single trip state, not a cached video segment.

Further Reading

• RFC 8216 and lesson 51, design a video streaming platform, for the generation and token rules Atlas reuses.
• Lesson 23, methodology; lesson 24, capacity; lesson 26, load balancing; lesson 27, caching; lesson 29, replication; lesson 31, consistency.
• AWS Well-Architected Framework, reliability and performance pillars, used as questions against this failure table.
• Vendor documentation for the CDNs you contract: token validation, origin shielding, and purge. The design uses those published capabilities and does not depend on an unpublished control API.
• Primer for contrast: /learn/system-design/netflix-video-streaming.
• Next case study: design a ride booking platform.