Design a Cloud File Storage System

Advanced System Design & Software Architecture · Lesson 58 · Real-World System Design Case Studies

Lesson Overview

A learner on a train starts a 400 MB assignment upload, loses the tunnel for thirty seconds, and retries. A design that begins the object again from byte zero will fail that learner all evening. A design that marks the file "available" when the metadata row commits, before the last chunk is durable, will hand a classmate a download that stops halfway. TechLearningPro Drive has to do both jobs: move bytes reliably, and tell the truth about whether those bytes are complete.

Drive stores course assets, instructor drafts, and learner submissions. It is two systems that must not be confused. The metadata service owns names, folders, versions, authorization, and upload sessions. The object store owns immutable blobs. Metadata without a blob is a broken file. A blob without metadata is an orphan we can garbage-collect. This lesson keeps metadata in PostgreSQL, puts bytes in an object store that publishes a durability target, and uses the open tus resumable-upload protocol between client and Drive so a retry continues at Upload-Offset instead of starting over. Internally, large objects use the object store's documented multipart upload. When the store is Amazon S3, that is CreateMultipartUpload, UploadPart, CompleteMultipartUpload, and AbortMultipartUpload as S3 documents them. The course does not invent a storage vendor's private API.

Downloads of public, versioned course assets go through a CDN with a URL that changes when the version changes. Private submissions never sit on a public cache key. After authorization, the API returns a short-lived pre-signed request from the object store's documented mechanism, or streams the bytes itself for small files that must be scanned or watermarked. Design a Social Media News Feed stores media ids that point here. Design a Multi-Tenant SaaS Platform supplies the tenant boundary: every object key and every metadata row carries a tenant, and deduplication does not cross it.

Durability is a split promise. The object store's published durability covers stored blobs. Our promise covers the metadata that makes those blobs findable, the checksum that proves the version is complete, and the ability to resume, delete, and audit. Module 6 will ask which failure loses bytes, which failure loses only a name, and how you restore. Disaster Recovery and Business Continuity and Architecture Performance and Capacity Review are where those numbers get defended.

Learning Objectives

By the end of this lesson, you should be able to:

1. Split metadata and blob storage, and state the failure each one can have without the other.
2. Specify a resumable upload with chunk offsets, checksums, and a completion rule.
3. Choose between proxying bytes and issuing pre-signed part URLs, including malware scanning.
4. Design folders, versions, and conflict detection with a base version.
5. Explain durability, replication, and backup for metadata versus the object store's published target.
6. Place a CDN in front of public versioned assets and keep private objects off shared caches.
7. Estimate ingress, egress, and metadata growth for a learning workload.
8. Decide where content-hash deduplication is safe and where it becomes an existence oracle.
9. Delete and erase data, including orphan parts and CDN copies of public versions.
10. Authorize every download and every upload session, scoped to a tenant and a principal.
11. Review a design that confirms the upload before the checksum matches.
12. Carry the case into a Module 6 interview with a clear RPO story for metadata and blobs.

Prerequisites

• Lesson 24: capacity as arithmetic. Ingress bytes per day is the input, not "a lot of files."
• Lesson 27: CDNs cache immutable objects. Versioned URLs are the invalidation strategy.
• Lesson 29: synchronous metadata replica and what RPO 0 does and does not cover.
• Lesson 37: idempotent completion of an upload. Completing the same session twice returns the same version.
• Lesson 44: service identity when the feed or the assignment service asks Drive for a file.
• The tus resumable upload protocol, which this lesson uses as the client contract: creation, offset, and patch. You do not need to memorize every header before reading; the walkthrough names the ones we depend on.
• Lesson 57's rule that a media id is not a public URL.

Detailed Core Concepts

Two stores, two lifecycles

A file is the stable identity a user sees: a name, a parent folder, an owner, a current version. A version is an immutable blob: size, checksum, object key, state. An upload session is the in-progress version. Readers only see versions in state AVAILABLE. A session in UPLOADING is invisible to classmates even if some parts have already landed in the object store.

This split is the design. Updating a file does not overwrite bytes in place. It adds a version and, when the upload completes and the base version still matches, moves files.current_version_id. Object stores are built for immutable puts. Fighting that with random-access writes inside a blob is how partial updates corrupt downloads.

Functional requirements

Non-functional requirements

If a stakeholder asks for eleven nines because a cloud brochure said so, the answer is: that figure, when the vendor publishes it, applies to objects stored in their system after a successful complete, not to our metadata, not to in-progress parts, and not to a bug that marks a file available early.

Resumable upload

tus, as published at tus.io, is the client protocol:

• POST creates an upload. The client sends Upload-Length and optional metadata. The server returns 201 and a Location.
• HEAD returns Upload-Offset, the next byte the server will accept.
• PATCH sends a contiguous chunk with Upload-Offset. The server persists those bytes and responds with the new offset.
• The client retries PATCH or HEAD after a timeout. It does not start a second POST unless it intends a different version.

Drive's implementation stores the session row first, then writes bytes. A PATCH that arrives with an offset the server has not acknowledged is 409. A PATCH that replays an offset already acknowledged returns the current offset without writing the bytes again. That is the idempotent chunk. Chunk size in this design is 8 MiB, aligned to a size the object store accepts as a multipart part (S3's documented minimum part size, except the last part, is 5 MiB; 8 MiB sits above that floor). The last patch may be shorter.

Each part stores an MD5 or SHA-256 as the object store requires for that part, and the session stores a client-declared SHA-256 of the full object. Complete verifies the full checksum. If the store can only checksum parts, the server verifies the concatenated checksum as it streams to a temporary assembly, or it trusts a composite the store documents. Do not mark AVAILABLE on "all parts returned 200" without the full-object check the product promised.

Direct-to-store variant: for objects above 32 MiB, POST still creates the session, and the response includes pre-signed part URLs from the object store's documented presign operation. The client uploads parts to the store, then calls POST /v1/uploads/{id}/complete with the list of part etags. Drive verifies with the store's documented complete call and the checksum. Bytes do not pass through Drive's CPU. The cost is that malware scanning happens after the bytes land, and the version stays SCANNING until the scanner allows AVAILABLE. Submissions that must not be downloadable by peers stay in SCANNING or REJECTED. Public course assets uploaded by trusted instructors may skip scanning by policy; learner submissions may not.

Versions and conflicts

Two instructors editing one outline is a real event. Each complete carries base_version_id. If it equals files.current_version_id, we advance the head. If it does not, we still keep the uploaded version, we do not move the head, and we return 409 with both version ids. The client uploads were not wasted. The user chooses "keep mine as the new head" (a second call that sets the head if they have permission) or "discard." There is no silent last-writer-wins on the head pointer. Object bytes are immutable either way, so "discard" only drops the pointer and eventually the blob.

Folders are rows with parent_id. Renames change metadata only. Moving a file does not copy the blob. Cycles are rejected with a parent walk or a materialized path that forbids a folder becoming its own descendant. Listing is WHERE parent_id = ? ORDER BY name, file_id with a cursor, not a deep recursive crawl, on the hot path. A breadcrumb walk is a few primary-key reads.

Deduplication

SHA-256 of the plaintext, scoped to one tenant, can point two files at one blob and save money on duplicated course kits. It is also an oracle: if tenant B can learn that a hash already exists in tenant A, they learn that A holds that file. Cross-tenant dedup is out of the design. Within a tenant, dedup is optional and off for any object encrypted with a per-object data key and a random nonce, because identical plaintext no longer matches. The default in this lesson: no application-level dedup. The object store may compress. We accept duplicate blobs. The existence oracle and the encryption interaction are worse than the disk bill at our size. The trade-off section says when to turn dedup on.

Deletes and erasure

Soft delete sets deleted_at and hides the file from listings. Restore clears it. A lifecycle job, after the retention the tenant contracted (for example 30 days), deletes the blob via the store's documented delete and then the metadata. A legal hold stops the lifecycle. Erasure for a privacy request deletes every version's blob, the upload sessions, and the metadata, and writes an audit row that names the file id and the actor, not the body. Public CDN objects are versioned URLs; erasure of a public asset requires a CDN invalidation of that URL, which is best-effort and fast if the URL contains the version id and you stop publishing it. Do not reuse a version id.

Orphan parts: a client that never completes leaves multipart uploads. A daily job calls the store's documented abort for sessions older than 7 days and removes the session row. Incomplete bytes are not a version and are not downloadable.

Architecture and Design Diagrams

Clients speak tus or a complete call to Drive. Drive is the only component that decides AVAILABLE. The object store and the CDN never see the folder tree.

mermaidCopy
flowchart LR
  subgraph clients [Clients]
    App[Learner and instructor apps]
  end

  subgraph edge [Edge]
    LB[L7 load balancer]
    CDN[CDN for public versioned URLs]
  end

  subgraph drive [Drive]
    API[Drive API]
    Scan[Scan workers]
    GC[Lifecycle and orphan GC]
    DB[(PostgreSQL metadata)]
  end

  subgraph storage [Object storage]
    Obj[Object store blobs and parts]
  end

  App --> LB --> API
  API --> DB
  API --> Obj
  App -->|Large parts via pre-signed URL| Obj
  Scan --> Obj
  Scan --> DB
  GC --> Obj
  GC --> DB
  App --> CDN
  CDN --> Obj

mermaidCopy
sequenceDiagram
  participant C as Client
  participant D as Drive API
  participant DB as PostgreSQL
  participant O as Object store

  C->>D: POST upload Upload-Length and base_version_id
  D->>DB: Insert session UPLOADING
  D->>O: Create multipart upload
  D-->>C: 201 Location
  loop Each 8 MiB chunk
    C->>D: PATCH Upload-Offset
    D->>O: Upload part
    D->>DB: Advance acknowledged offset
    D-->>C: New offset
  end
  C->>D: POST complete with checksum
  D->>O: Complete multipart
  D->>D: Verify checksum
  alt Head still matches base version
    D->>DB: Version AVAILABLE and move head
  else Conflict
    D->>DB: Version AVAILABLE unattached
    D-->>C: 409 with both version ids
  end

For the direct-to-store path, the PATCH loop goes to the object store with pre-signed URLs, and Drive only sees complete. The offset authority for that path is the store's list-parts operation, which Drive queries on HEAD so the client still has one place to ask.

Technical Explanation

Capacity

Assumptions:

• 5 million stored files, average object 4 MiB, plus a long tail of video lectures up to 2 GiB.
• New bytes: 500,000 uploads per day at 4 MiB average ≈ 2 TiB/day ingress. Video days are bursty; design the pipe for 5 TiB/day.
• Downloads: 20 TiB/day egress, of which 90% are public course assets served from the CDN. Origin egress ≈ 2 TiB/day.
• Peak concurrent uploads: 200 large sessions and 50 small completes per second.
• Metadata row ~1 KB per version. 500,000 versions per day ≈ 0.5 GB/day of metadata before indexes. A few years fit one partitioned PostgreSQL table.

The network bill and the object-store request rate dominate, not metadata CPU. A complete of an 8 MiB-part file of 400 MiB is 50 part requests. At 200 concurrent 400 MiB uploads finishing in ten minutes, part rate is on the order of 200 * 50 / 600 ≈ 17 part operations per second average during the burst, with spikes when clients align. That is comfortable for a mainstream object store and uncomfortable if every part is proxied through a small API fleet that also serves listings. This arithmetic is why large parts go direct to the store. Small files (under 32 MiB) proxy through the API so we can checksum in-process and keep the client protocol one place.

Cache does not reduce ingress. The CDN reduces origin egress for public immutable versions. A cache of metadata for hot public files (the current version id of a popular course asset) cuts listing and download-auth reads. Private download authorization hits the primary or a replica that is within a freshness bound you accept; a stale "allowed" cache after a permission revoke is a security bug. Permission caches, if any, TTL in seconds and are explicitly deleted on revoke.

APIs we own

POST /v1/files creates a file and the first upload session.

jsonCopy
{
  "parent_id": "fld_12",
  "name": "assignment-3.pdf",
  "content_type": "application/pdf",
  "size_bytes": 419430400,
  "sha256": "<hex of the full object>",
  "base_version_id": null
}

Response 201 with file_id, upload_id, and Location for tus. If size_bytes exceeds 32 MiB, the response also lists pre-signed part URLs and part size. Pre-signed URLs expire in 15 minutes. The client refreshes them with POST /v1/uploads/{id}/parts rather than receiving a day-long signature.

HEAD /v1/uploads/{id} returns Upload-Offset and state.

PATCH /v1/uploads/{id} for proxied chunks, tus headers Upload-Offset and Content-Type: application/offset+octet-stream.

POST /v1/uploads/{id}/complete is idempotent. Same session, same checksum, returns the same version_id. A different checksum on a completed session is 409.

GET /v1/files/{id} returns metadata. GET /v1/files/{id}/content authorizes and either redirects to a pre-signed GET that expires in 5 minutes or streams the body. The redirect URL is not logged. GET /v1/folders/{id}/children?cursor= lists names.

DELETE /v1/files/{id} soft-deletes. POST /v1/files/{id}/erase is a privileged erasure. Public publish is an explicit POST /v1/files/{id}/publications that allocates a versioned CDN path; it fails for files in a private submission folder.

Service callers, such as the feed, pass a service credential and a file id. Drive checks that the file is AVAILABLE and that the service is allowed to read that visibility class. The feed does not receive the object-store key.

Schema

sqlCopy
CREATE TABLE files (
  file_id            uuid PRIMARY KEY,
  tenant_id          uuid NOT NULL,
  parent_id          uuid,
  name               text NOT NULL,
  owner_id           uuid NOT NULL,
  current_version_id uuid,
  deleted_at         timestamptz,
  created_at         timestamptz NOT NULL,
  UNIQUE (tenant_id, parent_id, name)
);

CREATE TABLE versions (
  version_id    uuid PRIMARY KEY,
  tenant_id     uuid NOT NULL,
  file_id       uuid NOT NULL,
  size_bytes    bigint NOT NULL,
  sha256        bytea NOT NULL,
  object_key    text NOT NULL,
  state         text NOT NULL,
  created_at    timestamptz NOT NULL
);

CREATE TABLE upload_sessions (
  upload_id           uuid PRIMARY KEY,
  tenant_id           uuid NOT NULL,
  file_id             uuid NOT NULL,
  base_version_id     uuid,
  size_bytes          bigint NOT NULL,
  sha256              bytea NOT NULL,
  offset_bytes        bigint NOT NULL DEFAULT 0,
  multipart_upload_id text,
  state               text NOT NULL,
  expires_at          timestamptz NOT NULL
);

CREATE TABLE publications (
  publication_id uuid PRIMARY KEY,
  tenant_id      uuid NOT NULL,
  version_id     uuid NOT NULL,
  cdn_path       text NOT NULL UNIQUE,
  created_at     timestamptz NOT NULL
);

object_key looks like {tenant_id}/{version_id} and is unguessable because version_id is a UUID. It is not a user-supplied path. Path traversal does not exist if the key is never concatenated with client input. The unique (tenant_id, parent_id, name) applies to live files; a partial unique index WHERE deleted_at IS NULL allows a new file to reuse a name after soft delete. Check constraints: offset_bytes <= size_bytes, and state = 'AVAILABLE' implies a non-null checksum match recorded by the application. PostgreSQL holds metadata. It does not hold a bytea of the file.

Scale, cache, and load balancing

The Drive API is stateless behind an L7 load balancer. Upload PATCH traffic for small files is heavy on bandwidth, so those instances are in a pool separate from listing and download-authorization if one saturates the NIC. Direct-to-store uploads never touch that NIC. Health checks /ready against PostgreSQL. There is no sticky session; the offset lives in the database, so any instance can answer HEAD.

Redis is optional: a cache of hot file metadata and a single-flight lock around complete so two completes do not double-move the head. The lock is an optimization. The transaction UPDATE files SET current_version_id = ? WHERE file_id = ? AND current_version_id IS NOT DISTINCT FROM ? is the correctness control. Complete runs that update.

CDN: public cdn_path is /pub/{tenant}/{version_id}/{slug}. The slug is cosmetic. The version id makes the object immutable, so the cache TTL can be long (a day, or a year) and a new version is a new path. Do not purge the whole tenant prefix on every edit. Private content uses pre-signed URLs with cache-control private, no-store and is not given a publication row. If a CDN sits in front of pre-signed URLs, the cache key must include the signature and you still prefer not to cache them. The safe pattern is origin fetch with private.

Object-store request rate: list and download authorization do not call the store. Only upload, complete, pre-sign, and GC do. Pre-sign is a local HMAC with a key from the secret manager when the store documents that algorithm; it does not need a store round trip on every download if the documented signature is computed locally. S3 query-string authentication works that way. The signing key is as sensitive as the blob itself. It does not go in the metadata database.

Security

Every method checks tenant and an authorization policy: owner, folder grant, or public publication. Grants are rows (file_id or folder_id, principal_id, role) evaluated in the API, not by obscurity of the object key. Guessing a UUID is not permission.

Pre-signed URLs are capability URLs. Keep them short, do not log them, and do not put them in referrer headers to third parties if you can avoid it. The browser downloads directly from the store host; configure the store's documented CORS policy for the Drive web origin only.

Encryption: the object store's server-side encryption covers disks the vendor manages. A tenant that requires its own key uses envelope encryption: a data key per version, wrapped by a tenant key in the cloud KMS's documented wrap operation. Drive stores the wrapped key next to the version. Losing the tenant key makes the blobs unreadable, which is the point, and it means dedup across keys is impossible. Scan workers decrypt only inside the scan environment.

Malware scan is asynchronous. The threat is a submission that becomes AVAILABLE to other learners before the scan. The state machine forbids that transition. Instructors downloading their own upload may be allowed in SCANNING so they can confirm the file; peers are not.

Path and archive bombs: a zip submission is stored as a blob. A preview feature that unpacks archives must cap expanded size and reject nested depth. That preview worker is not required for the storage design, and it is a common way object storage gets turned into a compute incident. If you add previews, isolate them.

Failure modes

The dangerous window is "store complete succeeded, metadata not committed." The client sees an error and retries complete. Complete must be idempotent on the multipart id: if the object exists and the checksum matches, write the version row and return it. That repair is the same code as the happy path, not a manual script.

Trade-offs

Proxy versus pre-signed parts. Proxy keeps checksums and scanning simpler and burns API bandwidth. Pre-sign scales to video and moves malware scanning after the fact. Use the size threshold. Do not pre-sign a URL that allows PutObject on an arbitrary key; sign the exact part and the exact key.

One region versus cross-region replication of blobs. Cross-region copies survive a regional object-store outage and cost money and replication lag. Metadata already has a second zone. For ordinary assignments, one region plus the vendor's multi-AZ durability is enough. For the catalog of record of a paid course, enable the store's documented replication on that bucket prefix and say so in the SLO. Do not imply every learner upload has a second region.

Dedup on versus off. Off by default. Turn it on within a tenant for a bucket of plaintext-identical kits, with the hash stored as a blob pointer, and never as a signal you return to a different tenant ("this hash already exists" is the oracle).

Last-write-wins versus 409. Last-write-wins is fewer support tickets and silent data loss. Conflict return is more UI work and no silent loss. Drive uses 409 on the head pointer. Real-time collaborative editing of the same bytes is a different product (operational transform or CRDT) and is out of scope. Do not pretend versions are Google-Docs-style live collaboration.

Soft delete versus immediate erase. Soft delete saves instructors from mis-clicks. Immediate erase is what a privacy request and a malware reject need. Both exist. Only one is the default for the delete button.

Practical Implementation or Design Walkthrough

Assignment upload on a flaky network. The client hashes the file, calls POST /v1/files with the size and hash, and receives an upload id. It PATCHes 8 MiB at offset 0. The server writes part 1, sets offset_bytes = 8388608, and commits. The tunnel drops during part 4. The client HEADs, receives the offset, and continues. It does not create a second file. When the last byte is acknowledged, complete runs the store's complete, recomputes or confirms SHA-256, and moves the head because base_version_id was null on a new file. The assignment service sees AVAILABLE and allows the grader to fetch content. Until that state, the grader's GET returns 404.

Two instructors, one outline. Both start from version v1. Instructor A completes first; the head becomes v2. Instructor B completes with base_version_id = v1. Drive stores v3 as unattached and returns 409 with current = v2 and uploaded = v3. B's client downloads v2, the human merges offline, and uploads v4 based on v2. Nothing in v2 was destroyed by B's upload.

Public lecture. An instructor publishes version v7. Drive inserts publications.cdn_path. Learners' video players use that URL. A later fix is v8 and a new path. Cached v7 may remain until its TTL for anyone who has the old URL; the course page points at v8. Erasure of v7 invalidates /pub/.../v7/... and deletes the object.

Direct-to-store complete, the idempotent form:

javaCopy
public Version complete(UUID uploadId, byte[] expectedSha256) {
  Session session = sessions.lock(uploadId);
  if (session.state().equals("COMPLETED")) {
    if (!Arrays.equals(session.sha256(), expectedSha256)) {
      throw new ConflictException("checksum");
    }
    return versions.get(session.versionId());
  }
  objectStore.completeMultipart(session.objectKey(), session.multipartUploadId(), session.parts());
  byte[] actual = objectStore.headChecksum(session.objectKey());
  if (!Arrays.equals(actual, expectedSha256)) {
    objectStore.delete(session.objectKey());
    sessions.fail(uploadId);
    throw new ChecksumException();
  }
  return sessions.markAvailable(session);
}

headChecksum stands in for the store's documented header or a checksum you computed while proxying. If the store does not return a full-object SHA-256, the proxy path computes it as bytes stream, and the direct-to-store path requires the client hash plus a server-side recomputation from a ranged read or from part checksums composed in a way you can defend. Do not skip the compare. markAvailable is the conditional head update in one transaction. Tests: retry complete once; mismatch deletes the object and leaves the previous head; 409 when the base version moved; HEAD offset unchanged after a replayed PATCH.

Enterprise Real-World Scenarios

Scenario 1: A university's exam scripts and a public course catalog

The same Drive deployment holds public lecture videos and private exam scripts. A well-meaning change puts a CDN in front of the entire bucket to cut the egress bill. Exam PDFs are cached on shared edge nodes. The fix is prefix and API separation: only rows in publications are CDN-eligible, and the bucket policy, as the store documents bucket policies, denies anonymous GetObject except on the public prefix. Private keys require a signature. The bill goes up relative to the mistaken cache and stays far below origin-egress of the video catalog, because the videos were the bytes. The exam scripts were the risk. Capacity planning separates those two prefixes so a finance chart cannot argue for making the private prefix public.

The university also asks for a 15-minute RPO. Metadata already meets it with synchronous replication. Blobs meet the object store's durability, which is not the same sentence as RPO of in-progress uploads. Incomplete exam uploads are resumable and are not "lost data" if the session row survived. The disaster-recovery drill deletes a primary metadata node, promotes the replica, resumes an in-flight tus session from HEAD, and completes it. That drill is the evidence Disaster Recovery and Business Continuity asks for, not a screenshot of a replication setting.

Scenario 2: A corporate tenant brings a customer-managed key

A company tenant requires that TechLearningPro cannot read submission bytes without an action in the company's key system. Envelope encryption with a tenant key in a KMS the company controls meets that requirement when the wrap and unwrap calls are the cloud's documented API and the company can disable the key. Drive stores wrapped data keys on the version row. Scan workers in TechLearningPro's network cannot decrypt when the key is disabled, so malware scanning for that tenant runs in an environment the tenant accepts, or the tenant waives scanning in the contract. The design does not quietly fall back to an unencrypted copy "so the scanner works." Dedup stays off for this tenant. A support tool that downloads the blob for "debugging" is a policy violation and a missing audit event; the tool calls the same authorized download path and writes the audit row.

This tenant shares the metadata database with others, so row-level tenant predicates from lesson 60 apply. The blob prefix is the tenant id. A bug that signs a URL for a key without checking the tenant is the incident. The test suite includes a second tenant and expects 404 on the foreign file id and no pre-sign call.

Best Practices

• Keep bytes out of PostgreSQL. Keep authorization out of the object key's secrecy alone.
• Acknowledge offsets only after the part is durable, and make part uploads idempotent by part number.
• Promote a version to AVAILABLE only after checksum verification and, for untrusted uploads, after scanning.
• Use base_version_id and a conditional head update. Return 409 rather than overwriting.
• Version public CDN URLs. Cache immutably. Invalidate a path on erasure, not the whole site.
• Pre-sign the exact key and part, with a short expiry. Do not log the URL.
• Separate API pools if proxied uploads can starve metadata reads.
• Abort stale multipart uploads on a schedule.
• Scope dedup, when it exists, to a single tenant, and never confirm a hash's existence outside that tenant.
• Include tenant_id on every row and in every object key.
• Drill metadata failover with an in-flight resume, not only with a restored backup.
• Refuse client-supplied object keys and folder paths that escape the parent chain.
• Audit erasure and privileged downloads. Do not audit by storing the file body in the log.

Common Mistakes and Anti-Patterns

Available before durable. Setting the head pointer when the HTTP upload handler returns, before the store acknowledges the last part, produces downloads of truncated files. The state machine waits for complete and checksum.

Restart from byte zero. Clients that catch an error and call POST /v1/files again fill the bucket with prefixes and never finish on a bad network. Persist the upload id on the client. Resume with HEAD.

Public bucket, private metadata. The database denies the listing, and the bucket allows GetObject for anyone with the key. Keys leak through logs, referrers, and browser history. Default-deny on the bucket. Grant with short signatures.

CDN in front of everything. It saves money and publishes exam scripts. Publications are an explicit row.

Last-write-wins on a shared outline. The slower instructor's hours disappear. Conflict detection is a conditional update.

Cross-tenant dedup. It saves a copy of a popular textbook pdf and tells every tenant whether any other tenant uploaded that exact file. Do not build the oracle.

Logging the pre-signed URL. The log system becomes an access path that bypasses revocation. Log the file id and the actor.

Unpacking archives on the API server. A zip bomb becomes an availability incident. Store the blob. Scan and preview in an isolated worker with hard caps, or not at all in version one.

Orphan multiparts with no GC. Incomplete video uploads become the majority of the bill. Abort them after the session expiry.

Troubleshooting and Design Review

Design review for Module 6:

• Where is the byte acknowledged, and where is the version visible?
• What is the RPO of metadata, of completed blobs, and of in-progress parts?
• Which URLs may be cached, and which bucket prefix is anonymous?
• How does a retried complete behave after a crash in the dangerous window?
• How does tenant id constrain keys, pre-sign, and dedup?
• What is the GC story for parts and soft deletes?

Interview Questions and Answers

Q1. Why split metadata and blobs?

PostgreSQL is good at transactions, unique names, authorization queries, and conditional head updates. It is a poor store for a 2 GiB video and for 2 TiB of new bytes a day. An object store is good at immutable blobs, multipart puts, and a published durability number, and it is a poor store for "list this folder for this user" or "compare base versions." Split them. The file row points at a version, the version points at an object key, and AVAILABLE means both the row and the checksum passed. A blob without a row is garbage. A row without a blob is a bug you detect with a head request and repair. One database bytea column for "simplicity" fails the capacity test and the backup test on day one.

Q2. How does a resumable upload work when the client and the server both retry?

The client uses tus: create once, persist the Location, HEAD the offset, PATCH from that offset. The server commits offset_bytes only after the object store accepts the part. A replayed PATCH for an old offset returns the current offset and writes nothing. Part numbers make the store-side upload idempotent. Complete is idempotent on the session: a second complete with the same checksum returns the same version, including the case where the object was finished and the metadata commit failed the first time. A new POST is a new version, not a resume. The client's job is to remember the session. The server's job is to make HEAD and complete safe.

Q3. When do you proxy bytes, and when do you pre-sign?

Under 32 MiB, proxy. The API computes the running checksum, the volume is manageable, and scanning can happen before AVAILABLE without a second read. Above that, pre-sign each part for a short time for the exact key, let the client talk to the store, and complete in Drive after list-parts and checksum verification. Pre-signing PutObject to a client-chosen key is an authorization bypass. Scanning for the direct path is a state, SCANNING, not a comment. Peers cannot download until AVAILABLE.

Q4. What do you tell a stakeholder who quotes eleven nines?

I ask which document and which storage class. Vendors publish durability for objects after a successful put, often across multiple availability zones. That number is not the RPO of our metadata, not the fate of an unfinished multipart, and not a promise that a bug will refrain from deleting the object. Our metadata target is RPO 0 for committed transactions via synchronous replication, with an RTO we have drilled. Blob durability is the vendor figure for the class we buy, plus optional cross-region replication for the public catalog prefix. In-progress uploads are recoverable from the session row and the parts, which is resume, not magic durability. Non-Functional Requirements and SLO Design is where those three sentences stay written down.

Q5. How do conflicts work if two people upload the same file?

Each session records the base_version_id it started from. Complete advances current_version_id only if it still equals that base, in one conditional update. The loser's bytes remain a real version, unattached, and the API returns 409 with both ids. We do not overwrite the winner's pointer and we do not delete the loser's object automatically. The user merges and uploads a new version based on the winner. Live co-editing of one document is a different system. Version pointers are compare-and-swap.

Q6. How should the CDN be used?

Only for published, immutable, versioned paths of public assets. The path contains the version id, so the cache TTL can be long and an edit is a new URL. Private submissions use short-lived pre-signed GETs with private caching and no publication row. The bucket policy denies anonymous reads except on the public prefix. Invalidation is for erasure and legal removal of a specific path, not for routine edits. Putting the whole bucket behind the CDN is how private files leak. Home feeds and authorization decisions do not belong at the edge either; lesson 57 made the same point about rendered pages.

Q7. Where do you enforce authorization?

In Drive, on every metadata read, every upload, every complete, and every pre-sign. The object key is a UUID path under the tenant prefix, not a secret. A valid signature is a capability that Drive mints only after the policy check, and it expires in minutes. Folder grants and ownership live in PostgreSQL. The feed service uses its own credential and still passes through Drive; it does not get the bucket root key. Tenant id is part of the policy. A file id from another tenant 404s and does not pre-sign.

Q8. What is your delete and erasure story?

The user-facing delete is soft: deleted_at, hidden from listings, restorable until retention ends, then blob delete and metadata delete. Erasure skips the wait, deletes every version's object, aborts sessions, removes publications, invalidates CDN paths, and writes an audit record without the body. Legal hold blocks both the lifecycle and erasure. A malware reject deletes the blob and keeps an audit of the hash and the actor. GC separately aborts multipart uploads whose sessions expired. Forgetting GC is how the invoice grows without the file count growing.

Q9. Is content-hash deduplication worth it?

Not as a default. Within one tenant, identical course kits can share a blob and cut storage. Across tenants it is an existence oracle, and with per-object random encryption it stops working because ciphertext differs. The bug form is worse than the feature: returning "already uploaded" for a hash the caller does not own. I would ship Drive without dedup, measure the duplicate rate inside a tenant, and add a pointer from version to blob only if the bill justifies it. Checksums stay even when dedup is off. Checksums are integrity. Dedup is a cost optimization.

Q10. How do you size the system?

Start from uploads per day and average size. Five hundred thousand times 4 MiB is about 2 TiB of ingress per day. Egress is larger because course videos are watched repeatedly; put 90% of that on the CDN and size origin for the rest. Metadata is hundreds of megabytes per day and is not the bottleneck. Part-request rate on the object store, API bandwidth for proxied small files, and scan-worker throughput are the bottlenecks. Two hundred concurrent large uploads should be direct-to-store or they will consume the API fleet. Say the thresholds and the prices. Architecture Performance and Capacity Review expects this split of ingress, origin egress, and CDN egress.

Q11. A complete call crashes after the object store has finalized the object and before the database commit. What does the client do, and what do you do?

The client sees a timeout and calls complete again with the same upload id and checksum. The handler observes the object already present, verifies the checksum, and inserts the version if it is missing, then performs the conditional head move. If the head moved to this version on a commit that actually succeeded and the client merely lost the response, the second call sees COMPLETED and returns the same version. If the checksum does not match the object, we do not attach it. The repair is the normal complete path made idempotent, plus a reconciliation job that lists finalized multiparts with no version row. That job is the backstop when the client never retries.

Q12. What do you leave out of version one on purpose?

Real-time collaborative editing, cross-tenant dedup, client-supplied storage keys, anonymous bucket reads, unpacking zip files on the API, and a second regional copy of every learner upload. Each has a trigger to add it: product demand for co-editing, a measured duplicate bill inside a tenant, a contractual residency requirement. Leaving them out is how the state machine stays explainable. The capstone can add a retention-policy engine per tenant on top of this lifecycle. It should not replace tus, checksums, or the publication boundary.

Hands-On Exercises

Exercise 1: Specify upload, conflict, and download

Problem. Learners upload assignments up to 400 MiB on mobile networks. Instructors co-edit a public syllabus. Exam scripts are private. Lectures are public videos.

Tasks.

1. State integrity and resume requirements.
2. Draw the proxied path and the pre-signed path, with the size threshold.
3. Define session and version states, including scanning.
4. Show the conditional head update and the 409 payload.
5. Specify CDN eligibility and bucket anonymity.
6. Estimate daily ingress at 500,000 files times 4 MiB, and what fraction of egress hits the origin.
7. Describe GC for abandoned multiparts and soft deletes.
8. Describe the crash after the store finalizes and before the metadata commit.

Deliverables. State diagram, API list, capacity paragraph, and a threat note for a public bucket.

Exercise 2: Review a single-bucket design

The proposal stores bytes as bytea when small and in one public bucket when large, overwrites the object key on each edit, uses last-write-wins, logs the pre-signed URL for support, and deduplicates by SHA-256 globally. Write the review. Cover backup size, private-file exposure, lost updates, log leakage, and the cross-tenant oracle.

Rubric.

Lesson Summary

TechLearningPro Drive keeps names, authorization, and version pointers in PostgreSQL and keeps bytes in an object store addressed by tenant and version id. Clients resume through tus offsets. A version becomes visible only after checksum verification and any required scan. Concurrent edits conflict on the head pointer instead of overwriting it. Public assets are immutable CDN URLs. Private files are short-lived pre-signed requests minted after a policy check. Dedup across tenants is rejected. Abandoned parts are aborted. Metadata failover is drilled with an in-flight resume, and blob durability is quoted from the vendor's document rather than improved by adjectives.

The feed and the assignment product consume file ids, not bucket keys. Module 6's SLO, capacity, security, and disaster-recovery lessons are where you defend the three different durability stories: committed metadata, completed objects, and in-progress parts.

Further Reading

• Database Replication and Consistency, Caching Strategies and Cache Invalidation, and Idempotency and Exactly-Once Processing.
• Design a Social Media News Feed for media ids, and Design a Multi-Tenant SaaS Platform for tenant isolation of these rows.
• Disaster Recovery and Business Continuity and Architecture Performance and Capacity Review.
• tus resumable upload protocol: https://tus.io/protocols/resumable-upload.
• Amazon S3 multipart uploads, the documented internal mechanism when S3 is the configured store: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html.
• Amazon S3 presigned URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html.
• Amazon S3 FAQ, including published durability and availability by storage class: https://aws.amazon.com/s3/faqs/.