Design a Messaging System

Advanced System Design & Software Architecture · Lesson 50 · Real-World System Design Case Studies

Lesson Overview

TechLearningPro's cohort chat started as a thread of email. Instructors missed questions asked during a lab, learners pasted the same stack trace five times, and nobody could point at a message and say "this one, in this order." The replacement is Cohort Wire: direct messages between a learner and an instructor, and a channel per cohort that holds a few hundred members. It is not a global consumer messenger. The primer at /learn/system-design/whatsapp-messenger and the mock at /learn/system-design/mock-interview-chat-system sketch sockets, a store, and fan-out for a product with billions of devices. Those sketches skip the decision that actually shapes TechLearningPro's system: one ordered log per conversation, a read watermark instead of a row per reader, and a gateway that can die without taking the log with it.

This lesson applies the sequence from high-level system design methodology. Behavior and an explicit out-of-scope list come first, including end-to-end encryption and public group discovery, which this iteration does not offer. Estimates show that the hard number is concurrent connections and fan-out, not message inserts per second. The API distinguishes a durable accept from a device delivery. The schema gives each conversation a single writer for sequence numbers. The cache and the presence table are disposable. Idempotency shows up as a client message id. Queues show up as the bus between the store and the gateways. Sharding stays on the reopen list until a conversation's lock or a city's worth of history fills one primary.

Java and Spring appear as a TextWebSocketHandler at the edge and a transactional insert that allocates seq. The design does not invent a vendor chat API. WebSocket is RFC 6455. The frames we define are our contract.

Learning Objectives

By the end of this lesson you will be able to:

1. State the delivery guarantees Cohort Wire actually offers: durable ordering per conversation, best-effort push, and a read watermark.
2. Estimate message rate, history storage, fan-out, and concurrent sockets from labeled assumptions, and name which one forces a new component.
3. Design an API for send, sync, read watermark, and websocket attach, including idempotent retries.
4. Model conversations, membership, messages, and cursors so one conversation has one sequence authority.
5. Place gateways behind a load balancer without requiring sticky sends to succeed, using a location directory and a bus.
6. Explain presence as a TTL hint, and keep it off the authorization path.
7. Describe failure behavior when a gateway, the primary, or the bus is down, including what the sender's client must do.
8. Reject per-member message copies for cohort channels, and say when an inbox copy would earn its cost.

Prerequisites

• Lesson 23's sequence: scenarios before boxes. Lesson 49 is the previous case; its redirect cache is a different freshness problem from a message log.
• Lesson 26, load balancing. WebSocket connections are long-lived. The balancer's idle timeout and connection limits are part of the design, not an ops footnote.
• Lesson 31, consistency models. Per-conversation order is a real guarantee. Global order across channels is not.
• Lesson 33, message queues, for gateway fan-out. The queue is not the system of record.
• Lesson 36, distributed locking. Sequence allocation uses a row update in the conversation's database, not a lock service.
• Lesson 37, idempotency. Client retries after a timeout are the normal send path.
• Lesson 34's gateway ideas are useful later if mobile and web diverge. v1 uses one websocket contract.
• The consumer-messenger primers are vocabulary. Cohort size and durability claims in this lesson are tighter than those sketches.

Detailed Core Concepts

Product boundary

Actors are enrolled learners, instructors, and support staff who are members of a conversation. A conversation is either a direct pair or a cohort channel created when a cohort is published. Membership comes from enrollment. Wire does not let a learner invent a public room.

In scope:

• Send a text message to a conversation the caller belongs to.
• Persist it with a per-conversation sequence number.
• Push it to members who currently have a connection.
• Let any member sync history after a sequence number, so a reconnect or a new device catches up.
• Advance a per-member read watermark and show it to other members.
• Show a coarse presence hint: connected recently or not.
• Edit is out; delete-for-everyone is a tombstone in a later iteration and is named so we do not pretend the body can vanish from every backup.

Out of scope:

• End-to-end encryption. Messages are confidential in transit and at rest under company controls, and staff with a support role can be granted access under audit. A design that promises that instructors cannot read the cohort channel is a different product.
• Attachments, voice, and video calls. A later file service can store bytes; Wire would store a reference. Lesson 58 is the file case. This lesson does not smuggle object storage in.
• Public discovery, usernames searchable worldwide, and message requests from strangers.
• A global total order across conversations, typing indicators as a durable fact, and exactly-once device delivery.
• Push notification copy, badge counts on a locked phone, and email digests. Those belong to the notification platform in lesson 56. Wire emits "message persisted" and can hand off. The hand-off is a port, not a second messaging stack designed here.
• Groups of millions of subscribers. A cohort is hundreds of people. A school-wide broadcast is a different fan-out, recorded as a reopen condition.

Functional behavior

Send accepts a conversation id, a client-generated message id, and a text body up to 4,000 characters. If the caller is a member and the client id is new, the service commits a row with the next sequence number and returns that sequence. If the client id already exists in that conversation, the service returns the original sequence and body hash. If the hash differs, the response is 409. The sender's client renders the message when it sees the sequence, whether that sequence arrived on the socket it used to send or on the sync API after a timeout.

Sync returns messages with seq greater than a cursor, in order, up to a page size. Clients merge by seq. A gap means "ask again," not "invent an order."

Read receipt is PUT of last_read_seq for the caller. The server stores the maximum of the stored value and the offered value, so an old client cannot move the watermark backward. Other members learn the watermark through the same push path. We do not store "user A read message 12" as a row per reader per message. In a cohort of 200, a single busy day of 2,000 messages would be 400,000 receipt rows for one channel, and the product only needs "A has read up to 1,840."

Presence is a key with a TTL, refreshed by the gateway while the socket is open. Clients may show "active recently." They may not use presence to decide whether a message will be delivered. Delivery is "it is in the log." Push is an optimization.

Membership changes are events too. When enrollment removes a learner, the member row is closed. Later sync calls return 403. Messages already in the log stay. The removed member's client stops receiving pushes because the gateway checks membership on send and on fan-out.

Quality scenarios

Send durability. A member sends a message and receives 202 with a sequence, or a timeout. If they received the sequence, a later sync from any client of any member returns that message at that sequence. The measure is: after a successful response, 99.9 percent of syncs observe the row within 1 second, and 100 percent observe it once the primary acknowledges, excluding a region-loss disaster. We do not promise the message reached another human's screen inside that second.

Order. For one conversation, if the sender of message A received sequence 10 and the sender of message B received sequence 11, every sync returns A before B. There is no promise across two conversations. There is no promise that two in-flight sends from two people are ordered by wall clock. The sequence allocator is the clock that matters.

Reconnect. A client that drops and returns with after_seq equal to the last sequence it rendered receives only newer rows, without duplicates that would create a second bubble. Duplicate client ids collapse on the server. Duplicate pushes of the same sequence are harmless because the client keys bubbles by (conversation, seq).

Push latency. When the recipient has a live socket in the same region, p95 time from commit to frame on that socket is under 300 ms at the stated peak. If they do not have a socket, there is no push latency target. The notification port may send a mobile push; its SLO is that platform's, not Wire's.

Availability of send is 99.9 percent of accepted attempts in a month, where "accepted" means the primary committed. Gateway-only outages must not fail send if the client falls back to the HTTPS send API. That fallback is part of the design, not a nice-to-have.

Why one log, not an inbox copy

Two storage shapes compete in every messaging interview.

Per-conversation log. One row per message, ordered by seq, membership separate. Fan-out delivers a pointer. History is a range query. This matches cohorts, where many people must see the same transcript, including a learner who joins late and should see the lab notes from Monday.

Per-user inbox. On send, copy the message or a pointer into each recipient's mailbox. Reads are "my inbox, time order," which is lovely for a phone home screen and expensive when a cohort of 200 would multiply writes by 200. It also makes "the channel transcript" a reconstruction problem.

Cohort Wire uses the log. Each user still needs a list of conversations and an unread count. That list is derived: conversations where they are a member, unread as max(seq) - last_read_seq, with max(seq) cached on the conversation row. The home screen is not a second copy of every body. A celebrity-broadcast product, where one sender reaches millions who do not share a transcript, is where inbox copies or a publish-subscribe log with per-subscriber cursors earn their cost. We do not have that product. The reopen sentence is: if a single channel exceeds a few thousand simultaneous readers or the conversation row's update rate for last_seq becomes a hotspot, split fan-out into a dedicated bus and stop updating a counter on the hot row more often than the allocator already does. The allocator already updates one row per message. That is the hotspot to watch, and at cohort rates it is fine.

Sequence authority

Order requires a single decider per conversation. The decider is the SQL statement UPDATE conversation SET last_seq = last_seq + 1 WHERE id = ? RETURNING last_seq in the same transaction as the message insert. The row lock serializes senders of that conversation only. Other conversations proceed in parallel. A distributed lock around the same critical section would add a component that can fail independently and would not remove the need for the unique (conversation_id, seq) constraint. Lesson 36's guidance applies: prefer the constraint that matches the invariant.

The invariant is: one sequence, one body, one client id. Unique (conversation_id, client_msg_id) makes retries safe. Unique (conversation_id, seq) makes a bug in the allocator visible. Both constraints live in the same database as the conversation row so the transaction is local.

Connections are not the source of truth

A WebSocket is a session, RFC 6455, upgraded from HTTP and then idle for long periods with small frames. The gateway that holds the socket is a replaceable process. Send can arrive as a WebSocket frame or as POST /v1/messages. Both paths call the same write service. If the gateway dies, the socket dies, the client reconnects through the load balancer to any healthy gateway, and sync fills the gap. Sticky sessions make reconnect "usually" hit the same process and make deploys and crashes sharper. We do not require stickiness. The load balancer may use it as an optimization and must not promise it. Lesson 26's stateless rule still holds for the write path. The gateway is stateful only in the sense that RAM holds sockets. That state is a cache of "who is connected here," rebuilt on reconnect.

A directory maps user_id to gateway instance id with a TTL of about 45 seconds, refreshed with the heartbeat. On commit, the write service publishes a small event: conversation id, seq, and the member list or a request that gateways which care should pull. At our size the event can carry the body, because bodies are at most 4 KB and the peak message rate is modest. Gateways that have a local socket for a member write the frame. Gateways that do not, ignore it. If the directory says the user is on gateway G and G does not have the socket, the frame is dropped and the client will sync. Push is best effort. The log is the contract.

Presence, receipts, and honesty

Presence keys expire. A phone that loses radio looks offline within a minute, then online again, and the channel does not need a durable presence history. Storing every transition is a write amplifier with no scenario attached. The client shows "seen recently" from the TTL key and nothing more.

Read watermarks are durable because "I read up to here" is a user-visible fact used for unread badges. They are not a security boundary. A member who never advances the watermark simply looks unread. We do not infer consent or completion of a lesson from a watermark. That would couple chat to the learning record without a product decision.

"Delivered" is the word that causes the most design damage. If it means "the primary committed," the sender can see it when they receive seq, and no extra system is required. If it means "the recipient's phone displayed it," we do not know, and building a per-device acknowledgement for every message in a cohort of 200 multiplies traffic by the number of devices. Cohort Wire uses three words in the client: sent (client has the seq), in channel (it is in the log, which is the same moment), and read up to (watermark). The primer's double grey check is a consumer-messenger convention. We will not fake it with a weaker definition.

What the primers skip

/learn/system-design/whatsapp-messenger emphasizes fan-out and online status at a scale where the directory of sockets is the system. Cohort Wire's scale math, below, shows a few thousand messages per second at peak only if we accept a future multiplier we do not have today. Today's peak is a classroom problem: hundreds of sockets per cohort, a handful of messages per second in the busiest lab, and a primary that must not lose the transcript when a gateway restarts. The staff-level move is to refuse end-to-end encryption, inbox copies, and exactly-once push in the same breath as you draw the socket, and to show the transaction that allocates seq.

Architecture and Design Diagrams

mermaidCopy
C4Context
    title Cohort Wire — system context
    Person(learner, "Learner", "Sends and reads cohort and direct messages")
    Person(instructor, "Instructor", "Answers in the cohort channel")
    System(wire, "Cohort Wire", "Ordered message log, live push, read watermarks")
    System_Ext(idp, "Company IdP", "Authenticates members")
    System_Ext(enroll, "Enrollment", "Source of cohort membership")
    System_Ext(notify, "Notification platform", "Optional offline nudge")
    Rel(learner, wire, "Sends, syncs, holds a socket", "HTTPS and WebSocket")
    Rel(instructor, wire, "Sends, syncs, holds a socket", "HTTPS and WebSocket")
    Rel(wire, idp, "Validates tokens", "OIDC")
    Rel(wire, enroll, "Reads membership", "HTTPS")
    Rel(wire, notify, "Emits persisted-message hints", "queue")

mermaidCopy
flowchart LR
    client[Client]
    edge[TLS load balancer]
    gw[Gateway fleet]
    write[Message service]
    db[(Conversation log)]
    dir[(Connection directory)]
    bus[Fan-out bus]
    enroll[Enrollment]

    client --> edge
    edge --> gw
    edge --> write
    gw --> write
    write --> db
    write --> enroll
    gw --> dir
    write --> bus
    bus --> gw

The message service owns the log. Gateways own sockets. The bus is disposable. Enrollment remains the owner of who belongs in a cohort; Wire stores a membership projection it can refresh.

mermaidCopy
sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant W as Message service
    participant D as Primary
    participant B as Fan-out bus
    participant G2 as Other gateway

    C->>G: SEND client_msg_id, body
    G->>W: authorize and append
    W->>D: txn allocate seq and insert
    D-->>W: seq
    W-->>G: seq
    G-->>C: ACK seq
    W->>B: conversation, seq, body
    B->>G2: deliver
    G2-->>G2: write frame if local socket

Reconnect does not depend on that bus.

mermaidCopy
sequenceDiagram
    participant C as Client
    participant W as Message service
    participant D as Primary

    C->>W: GET messages?after_seq=42
    W->>D: select seq greater than 42 limit N
    D-->>W: rows in seq order
    W-->>C: page and next cursor

Technical Explanation

Send path

The client generates client_msg_id (a UUID is fine) before the first attempt and reuses it on every retry. The gateway or the HTTPS handler authenticates the bearer token, checks membership, and calls the write service. The transaction:

1. Insert into message_dedup or rely on the unique (conversation_id, client_msg_id).
2. If the insert conflicts, read the existing row and compare body hash.
3. Otherwise increment last_seq and insert the message with that sequence, the sender, the body, and the timestamp from the database clock.

Returning the database clock avoids a client-supplied timestamp becoming the order. Display order is seq. created_at is for humans and for retention jobs.

The response is 202 Accepted with seq once the commit returns. 202 is a bit of honesty: the server has the log entry and has not claimed that other devices have it. Teams who return 200 with "delivered" paint themselves into the double-check fiction. Either code is acceptable if the body says seq and the documentation refuses device delivery. This design uses 202 plus seq.

A timeout with an unknown outcome is the client's cue to retry the same client_msg_id. It must not mint a new id or the user will see two bubbles after sync. The user interface can show "sending" until either the ACK or a sync reveals the seq.

Fan-out path

After commit, publish one event per message, not one event per member. Each gateway consumes the bus. The gateway keeps an in-memory map from conversation id to local sockets, maintained on subscribe and on close. If the map is empty for that conversation, the gateway drops the event. This keeps the bus payload at one copy and keeps idle gateways cheap.

At a few hundred members with a handful online, this is easy. The failure mode is a gateway that lags the bus and then floods sockets, or a bus that redelivers. Redelivery is safe because clients dedupe on seq. A gateway must not "catch up" by inventing messages the log does not have. If it has been disconnected from the bus, it reconnects and clients sync. There is no gateway-side spool that outranks the database.

Member lists change. The gateway checks membership before writing a frame, using a short-lived cache of membership filled from the message service. A stale "still a member" entry can leak one message after removal until the cache TTL, which we set to 30 seconds. The scenario for removal is: within 30 seconds of membership close, pushes stop, and sync already returns 403. State that number in the record. Instant global disconnect of every socket is a harder promise and needs an explicit push of the membership event, which we do send as well. The TTL is the backstop when that push is lost.

Sync path and history

Sync is the API that makes the system correct. Push is the API that makes it feel alive. Any design review that only draws the socket has not designed messaging.

Pages are ordered by seq, limit 100, after_seq exclusive. The index is the primary key (conversation_id, seq), so the query is a range on one partition of the key. Clients store the highest seq they have rendered. When the socket opens, the client syncs before it trusts the socket, so a frame that races ahead of history can be held and inserted by seq. Holding one out-of-order frame is enough if a single gateway sends in order per conversation. After a reconnect to a different gateway, sync is what closes gaps. The client should sync on every connect, not only when it thinks it missed something.

History retention is 2 years for cohort channels, then a cold export if compliance asks, then deletion of the hot rows. Direct messages follow the same default unless a privacy request arrives. A privacy delete is a tombstone: body replaced, redacted = true, seq preserved so other clients do not see a gap they try to refill forever. Backups expire on their own schedule; the scenario says "interactive sync no longer returns the body," not "every backup tape was edited yesterday." Write that limitation down. Pretending otherwise is how teams design an unbuildable shredder.

Read watermarks

UPDATE conversation_member SET last_read_seq = GREATEST(last_read_seq, :offered) WHERE conversation_id = :c AND user_id = :u.

The watermark cannot exceed last_seq of the conversation. A client that offers a future seq is clamped. The update publishes a small event so other clients can move a marker. If that event is lost, the next sync of the conversation summary returns the stored watermarks. Summary is the repair path, same as messages.

Unread count for the home screen is last_seq - last_read_seq, computed when the client fetches summaries, not maintained as a counter that can drift. At our read rate of summaries this is cheap. A cache of summaries with a few seconds of staleness is optional and must not be the only place a new seq appears. Lesson 27: do not cache a fact tighter than its scenario. Unread badges may lag a few seconds. The transcript may not.

Presence and the directory

On socket open, the gateway writes presence:{user} and location:{user} with TTL 45 seconds and refreshes every 15 seconds while the socket lives. On clean close, it deletes both. On crash, TTL expires them. Two devices: location is a set of gateway ids, not a single value, or the second device steals the first and stops its pushes. A set is the difference between a demo and a learner who has a phone and a laptop. The event fan-out already does not need the directory if every gateway filters by local sockets. The directory exists so a future "push only to that user's gateways" optimization can avoid broadcasting every message to every gateway. At a fleet of four gateways, broadcasting is simpler and correct. The directory becomes worth it when gateway count makes irrelevant traffic measurable. v1 can broadcast on the bus to all gateways and keep the presence keys only for the green dot. Say so. An unused directory is a stale-data factory.

Scale, load balancing, and caching

The load balancer terminates TLS and forwards WebSocket upgrades. Idle timeout must exceed the client heartbeat (client ping every 20 seconds, server reads, balancer idle timeout at least 60 seconds). A default 30-second idle timeout will flap every lab session and look like an application bug. Connection caps per instance are explicit: plan for on the order of 50,000 sockets per gateway at a few kilobytes each, and add instances when the connection count or the file-descriptor limit says so. Deploys drain: stop accepting new upgrades, wait for clients to reconnect elsewhere, then exit. Clients already know how to sync. A deploy that kills sockets without a client retry loop is an incomplete client, and the client is part of the system.

Caching: membership for 30 seconds, conversation summaries for a few seconds, presence in the TTL store. Message bodies are not cached in front of the primary for sync. The range query is the source of truth, and a cache of "recent messages" goes stale in exactly the race reconnect cares about. At the QPS below, the primary can answer sync.

Sharding: not at v1. If we shard later, the key is conversation_id. All messages and the conversation row live together so the seq transaction stays local. Cross-conversation inbox queries are not our hot path. Lesson 30's scatter-gather is what we avoid by refusing per-user copies.

Security

Authentication is the company IdP on the upgrade and on every HTTPS call. The upgrade request carries the bearer token. A socket that has been open for hours re-checks the token on a timer or on the next send, so a revoked session does not talk until tomorrow. Interval: 5 minutes, and immediately on a failed send.

Authorization is membership on every send and sync. Conversation ids are unguessable UUIDs, and that is not the control. The control is the member row. A leaked URL in a screenshot must not grant access.

Body handling: store text. Clients escape on render. The server rejects NUL and rejects bodies over the limit. The server does not fetch URLs that appear in the body. Link previews would be SSRF, the same class of bug as lesson 49's forbidden unfurl.

Rate limits, lesson 35: per user, sends per minute, and a tighter limit for identical bodies. A stuck client retrying a new id every time is a bug we should surface; the dedup key only saves retries that reuse the id. A cohort-wide cap protects the seq row from a runaway bot. Excess sends get 429 with a retry-after. The client keeps the same client_msg_id.

Support access is a role that is audited per read, not a backdoor account that shares a member's password. The out-of-scope line on end-to-end encryption is what makes this role possible. If a later record adds encryption, support access has to be redesigned in the same record, not left as a quiet query.

Abuse: block a member from sending while leaving them able to read, by a flag on the member row. Removal is the enrollment event. We do not build a second social graph.

Failure handling

The send response waits for the database commit and does not wait for the bus. That is the availability boundary. Teams that wait for every gateway to ACK have built a distributed transaction by accident and will time out whenever one laptop's socket is slow.

Trade-offs

Log versus inbox copy. We take the log. Write amplification of cohort size is refused. The home screen computes unread from cursors.

Broadcast bus versus directed push. v1 broadcasts to a small gateway fleet. Directed push needs a correct multi-device directory. We add it when gateway count makes broadcast noisy.

Sticky sessions versus sync-on-connect. Stickiness reduces reconnect cost and increases blast radius of a dead node. Clients sync on connect, so stickiness is optional.

Database seq versus a single-threaded writer process. A writer process is easy to explain and becomes a bottleneck and a failover project. The row lock is the writer, and the database already fails over.

Durable presence history versus TTL keys. TTL keys flicker and do not fill disks. No scenario asked for history.

HTTPS send plus WebSocket versus socket-only. Socket-only send fails when the upgrade is blocked by a network and when the gateway is restarting. HTTPS send is the path that matches the durability scenario. The socket is for push and for low-latency send when it is up.

Practical Implementation or Design Walkthrough

Assumptions

Arithmetic

Average message rate:

[ \frac{500{,}000}{86{,}400} \approx 5.8 \text{ messages/s} ]

Peak message rate under B4:

[ 5.8 \times 10 \approx 58 \text{ messages/s} ]

Even if one cohort took the entire peak, the seq row would update 58 times per second. A single primary handles that. The estimate does not support a custom sequencer fleet.

Fan-out frames at peak, if we naively pushed to every member of a 300-person cohort for every message and every message were in such a cohort:

[ 58 \times 300 = 17{,}400 \text{ frames/s} ]

Under B6 only about 40 sockets are live, and not every message is in a max-size cohort. Live frames are closer to (58 \times 40 = 2{,}320) per second if the peak were one huge lab, which it is not. The fleet-wide number is still small for a bus. It is large enough that doing it as 300 synchronous HTTP calls inside the send transaction would be reckless. The bus exists because of the failure isolation, not because 2,000 frames per second requires a cluster.

Storage, 2 years:

[ 500{,}000 \times 365 \times 2 = 3.65 \times 10^8 \text{ messages} ]

[ 3.65 \times 10^8 \times 500 \text{ bytes} \approx 180 \text{ GB} ]

including indexes and overhead in the 500-byte planning figure. One primary with a replica holds this. Attachments would change the diagram; they are out of scope precisely so this number stays about the log and not about video.

Sockets: 20,000 concurrent. At roughly 50 KB of server-side overhead per socket including buffers, that is about 1 GB, fine on two gateways for redundancy, with a third during deploys. The saturated element at today's peak is not CPU. It is a single gateway's connection table during a deploy that forgot to drain, or a primary exhausted by a membership query without an index. Plan headroom for a bad lab day at 5 times the socket count, still inside a handful of gateways.

Summary reads: (50{,}000 \times 30 / 86{,}400 \approx 17) requests per second average, peak maybe 100. Indexed membership lookups. Cache optional.

Multiplier M2, labeled, not a forecast: 100 times the messages, 5 million per day, peak about 5,800 per second. A single conversation still serializes its own sends. If one conversation reached hundreds of sends per second, the seq row would be the hotspot and we would move that conversation's log to a dedicated partition or batch allocates. If the 100 times is spread across conversations, shard by conversation_id. Socket count at 100 times B2 is 2 million concurrent, which is a gateway fleet and a directory, still not a reason to copy message bodies per user.

API

POST /v1/conversations/{conversationId}/messages

• Auth required. Body: clientMsgId, body.
• 202 with seq, conversationId, clientMsgId.
• 200 acceptable on idempotent replay with the same payload; we use 202 for both the first commit and the replay so clients have one success shape, plus a header Idempotent-Replay: true on replay.
• 409 same client id, different body hash.
• 403 not a member.
• 413 body too large.
• 429 rate limit.
• 503 primary unavailable.

GET /v1/conversations/{conversationId}/messages?after_seq={n}&limit={n}

• 200 with messages in ascending seq and nextSeq if the page is full.
• 403 if not a member.

PUT /v1/conversations/{conversationId}/read

• Body: lastReadSeq. 204. Clamped and monotonic.

GET /v1/conversations

• Summaries: id, kind, last seq, caller's watermark, a preview of the last body. Preview comes from the log, not from a second store, unless a measured p99 says the extra query hurts. At 100 summary QPS it should not.

GET /v1/ws upgrade. First server frame is hello with the server time. Client sends sync cursors or uses HTTPS sync. Server pushes message, watermark, and membership frames. Client sends send, read, and ping.

Frame bodies are JSON objects with a type field. This is our contract. It is not STOMP and it is not a vendor protocol. If a team prefers STOMP over a broker, that is an implementation of the bus, and the log is still the database. Spring's documentation describes a broker relay for STOMP; adopting it does not remove the need for seq.

Schema

sqlCopy
CREATE TABLE conversation (
    id          uuid PRIMARY KEY,
    kind        varchar(16) NOT NULL,
    last_seq    bigint NOT NULL DEFAULT 0,
    created_at  timestamptz NOT NULL,
    CONSTRAINT conversation_kind_chk CHECK (kind IN ('direct', 'cohort'))
);

CREATE TABLE conversation_member (
    conversation_id uuid NOT NULL REFERENCES conversation(id),
    user_id         uuid NOT NULL,
    role            varchar(16) NOT NULL,
    joined_at       timestamptz NOT NULL,
    closed_at       timestamptz,
    send_blocked    boolean NOT NULL DEFAULT false,
    last_read_seq   bigint NOT NULL DEFAULT 0,
    PRIMARY KEY (conversation_id, user_id)
);

CREATE INDEX member_by_user ON conversation_member (user_id) WHERE closed_at IS NULL;

CREATE TABLE message (
    conversation_id uuid NOT NULL,
    seq             bigint NOT NULL,
    sender_id       uuid NOT NULL,
    client_msg_id   uuid NOT NULL,
    body            text NOT NULL,
    body_hash       bytea NOT NULL,
    redacted        boolean NOT NULL DEFAULT false,
    created_at      timestamptz NOT NULL,
    PRIMARY KEY (conversation_id, seq),
    UNIQUE (conversation_id, client_msg_id)
);

Direct conversations can add a unique pair of member ids in a side table so we do not open two rooms for the same two people. Cohort conversations are keyed by the cohort id stored in a column subject_id with a unique constraint. Enrollment is still the authority; this row is the projection.

Spring notes

The write service is a Spring transaction. RETURNING is PostgreSQL, which this case uses because the seq update and the insert must be one commit. If the company standard were another engine, the requirement would be the same: one local transaction, a conditional update, a unique client id.

javaCopy
@Transactional
public SendResult append(SendCommand cmd) {
    Member member = members.lockOpenMember(cmd.conversationId(), cmd.senderId());
    if (member == null) {
        throw new NotMemberException();
    }
    if (member.sendBlocked()) {
        throw new SendBlockedException();
    }
    Optional<Message> prior = messages.findByClientId(cmd.conversationId(), cmd.clientMsgId());
    if (prior.isPresent()) {
        if (!prior.get().bodyHash().equals(cmd.bodyHash())) {
            throw new IdempotencyConflictException();
        }
        return SendResult.replay(prior.get().seq());
    }
    long seq = conversations.allocateSeq(cmd.conversationId());
    messages.insert(cmd, seq);
    return SendResult.accepted(seq);
}

allocateSeq is the UPDATE ... RETURNING statement. findByClientId plus insert is safe under the unique constraint if two transactions race: the loser rolls back and the caller maps the integrity violation to a replay read. Map that violation inside the service so the controller does not return 500.

The gateway extends TextWebSocketHandler. On afterConnectionEstablished, register the socket in the local multimap and start heartbeats. On handleTextMessage, parse the type and call the same append method the HTTP controller calls. Do not maintain a second write path that "just writes to the socket peers" and forgets the database. On afterConnectionClosed, drop the multimap entries. Heartbeats that fail close the session.

Publish to the bus after the transaction commits, using a transactional outbox if a crash between commit and publish would bother you. For push, a lost publish is repaired by sync, so an outbox is optional. For the notification port, a lost "nudge" means a quiet phone, which may be worth an outbox. Prefer the outbox for the notification event and accept best-effort for the in-region bus, or use one outbox for both. One outbox is easier to explain. Lesson 33's outbox pattern is the tool. The consumer must be idempotent on (conversation_id, seq).

Tests

• Two concurrent sends receive distinct seq values and sync returns both in order.
• Retry with the same client id returns the same seq and does not increment last_seq.
• Retry with the same id and a new body returns 409 and does not change the row.
• Client disconnect after commit, reconnect to a different gateway, sync shows the message once.
• Bus consumer down: send still returns seq; after the consumer returns, push resumes; a client that synced in between dedupes.
• Non-member 403. Closed member 403 on sync even if they know the id.
• Watermark monotonicity: a lower lastReadSeq does not stick.
• Rate limit does not force the client to change client id.
• Removal stops push within the membership TTL in an integration test with a short TTL.

Records

ADR-050-1. One log per conversation, seq allocated in the conversation row's transaction. Rejected: per-recipient inbox copies; a dedicated lock manager. Reopen if a single conversation's send rate saturates the row or a product appears whose readers do not share a transcript.

ADR-050-2. Push is best effort; sync is authoritative. Rejected: holding the send response until every online member's gateway acknowledges. Reopen if a scenario with a numeric device-delivery SLO is actually funded, and then only for direct messages where the fan-out degree is two.

Enterprise Real-World Scenarios

Lab hour and a gateway deploy

Forty learners in a cohort are connected when a deploy restarts a gateway. Sockets drop. The balancer sends reconnects to the remaining instance, which now holds most of the sockets. If the client reconnects and paints only new socket frames, every learner misses the message the instructor sent during the restart and, worse, may apply a later frame without the gap. The client specification requires sync from the last rendered seq before enabling the input. The deploy procedure stops new upgrades, waits until the connection count drains or a deadline passes, then exits. Learners see a brief "reconnecting" state. The transcript stays ordered because nobody wrote it on the gateway.

The incident variant is a gateway that stays up but loses the bus. Sockets look healthy, presence stays green, and messages appear only after someone refreshes. The metric that catches this is bus consumer lag and "pushes written" per gateway, compared with commits. A green socket count is not a delivery probe. A synthetic member in each environment that sends and expects a frame on a second socket is the probe.

The stuck retry that forked the transcript

A mobile client times out at 2 seconds because the learner's network stalled, shows an error, and mints a new client id for the automatic retry. The first request commits. The second commits too. Sync shows two identical answers, and the instructor replies to both. Dedup cannot help if the id changes. The fix is in the client contract: one id for the lifetime of the draft, a long-lived "sending" state, and a sync before you tell the user it failed. The server can additionally reject a burst of identical bodies from one sender inside a few seconds with 429 and a body that says "retry the same clientMsgId," which nudges broken clients without refusing a learner who genuinely said "yes" twice. Document the difference. Support will ask.

A related enterprise case is the teaching assistant who is a member of 80 cohort channels on Monday morning. Summary fetch is 80 membership rows and 80 last-message lookups. That is a query shape, not a shard. An index on the member table and a limit on channels returned, with a clear "you are in more channels than the home screen will load," beats a new inbox pipeline. If product later wants a single stream of all mentions across cohorts, that is a new read model fed by the bus, a CQRS choice from lesson 42, built beside the log rather than instead of it.

Best Practices

• Treat sync as the correctness path and the socket as an accelerator. Review both sequences.
• Allocate order with a transaction on the conversation, and enforce it with a unique (conversation_id, seq).
• Require a client message id that survives retries. Test the timeout-after-commit case.
• Store one body per message in a cohort. Compute unread from watermarks.
• Keep presence as a TTL hint. Do not authorize from it and do not page history of it.
• Drain gateways on deploy. Set balancer idle timeouts longer than heartbeats.
• Check membership on send, sync, and push. Use a short TTL cache plus an explicit removal event.
• Respond to the sender after the database commit, not after fan-out.
• Escape text on render. Do not fetch URLs from message bodies on the server.
• Name the delivery words the client uses, and refuse words you cannot observe.

Common Mistakes and Anti-Patterns

Writing the message only to the other sockets. A gateway restart erases the lab transcript. The log is the product.

Per-member copies for a shared channel. Writes scale with members, storage scales with members, and "what did the instructor say" becomes a merge. Copies fit a mailbox product. This is a transcript product.

Exactly-once delivery as a slogan. The database commit is at-least-once from the client's point of view because the response can be lost. Idempotent client ids make the user-visible effect once. Device push is at-most-once unless the client syncs. Say the three paths separately.

A global order or a wall-clock order. Two regions' clocks will invert two sends. seq will not, inside one conversation. Across conversations, order does not matter for this product.

Sticky sessions as the recovery plan. The node that held the stickiness is the node that failed. Clients must sync on any gateway.

Durable presence and per-message read rows. They dominate storage and still flicker. Watermarks and TTL keys match the UI we can honestly build.

Blocking send on the notification provider. A push vendor incident then blocks the classroom transcript. The port is asynchronous.

Second write path inside the gateway "for latency." It will drift from the service that checks blocks, membership, and dedup. One append method.

Using presence to decide whether to store the message. Offline learners are why the log exists.

Troubleshooting and Design Review

Symptom: Duplicate bubbles after a flaky network, same text, two sequences.

Likely cause: client minted a new clientMsgId on retry, or the server unique constraint is missing and a race double-inserted.

Investigation: compare client ids on the two rows. Check whether the client logs show a regenerated id. Look for integrity-violation handling that returns 500 and triggers a new id.

Corrective action: persist the draft id. Map unique violations to replay. Add the concurrent-retry test.

Symptom: Messages exist in sync and never appear until refresh, sockets look connected.

Likely cause: bus consumer stuck, gateway subscribed to the wrong topic, or the client applies only socket frames and the tester is not refreshing — if sync has them and the live UI does not, the push path is the suspect.

Investigation: consumer lag, gateway push counter, a synthetic two-socket probe.

Corrective action: restart the consumer, fix the subscription, alert on probe failure. Do not "fix" it by writing a second log on the gateway.

Symptom: One cohort's sends time out while others are fine.

Likely cause: a hot seq row from a bot, a long transaction holding the conversation lock, or lock waits from a membership update that locked the parent row in the wrong order.

Investigation: database lock waits for that conversation id, send rate, body sizes.

Corrective action: rate-limit the sender, fix lock ordering, and only then consider partitioning that log. Read the slow statements before adding a cluster.

Symptom: A removed learner still receives messages.

Likely cause: membership cache TTL too long, removal event not published, or the gateway checks the cache and never the closed flag.

Investigation: closed_at on the member row, cache TTL, whether a removal event hit the gateway.

Corrective action: shorten TTL to the scenario, publish removal, and test it. Force a disconnect on removal if the scenario tightens.

Symptom: Unread badges climb and never drop, or drop below zero.

Likely cause: watermark updates failing silently, GREATEST missing so old clients rewind the cursor, or last_seq cached and stale in one direction.

Investigation: member row versus conversation last_seq for one user.

Corrective action: monotonic update, compute unread at read time, fix the client that sends a zero watermark on startup.

Design-review questions: What is a successful send allowed to claim? Where is order decided? What happens to a retry after a lost response? What does the client do on reconnect? What is persisted if a gateway dies mid-push? How does a 300-member channel avoid 300 body copies? What is presence allowed to mean? Which dependency can be down while send still commits?

Interview Questions and Answers

Q1. What delivery guarantee do you actually offer?

Once the service returns a sequence, that message is in the conversation log at that sequence and any later sync will show it. Push to online sockets is best effort and may arrive twice or late; clients dedupe on sequence. There is no promise that another person's device has rendered the message. Read state is a watermark per member, not a per-device acknowledgement. I use those words in the API so the client cannot imply a double check we did not build.

Q2. How do you keep order?

Each conversation has last_seq. The send transaction increments it and inserts the message with the returned value, under the row lock, with a unique key on (conversation_id, seq). All sync reads order by that key. I do not order by client timestamps. Two conversations are not ordered against each other. A lock service in front of the same update would add a failure mode without a stronger invariant than the unique key.

Q3. How do you handle the client retrying after a timeout?

The client creates one clientMsgId per draft and sends it on every attempt. The server's unique constraint returns the original sequence if the body hash matches and 409 if it does not. The user interface stays in "sending" until an ack or a sync result arrives. Generating a new id on timeout creates a duplicate message the constraint cannot see. That is the bug I test.

Q4. Why not copy the message into every member's inbox?

A cohort of 300 would multiply storage and writes by 300, and the transcript would have to be reconstructed. The product is a shared log plus a per-member cursor. Unread is last_seq - last_read_seq. I would switch shapes for a broadcast where recipients do not share a history and each has a private mailbox. That is a different product, and I would keep the log as the write model even then, with the inbox as a derived read model.

Q5. How do WebSockets and the load balancer interact?

The balancer terminates TLS, allows the upgrade, and must use an idle timeout longer than the heartbeat or it will drop quiet labs. Gateways are stateful only for the sockets in RAM. Send and sync work on any instance through the message service. Stickiness is optional. Deploys drain upgrades and let clients reconnect and sync. I size gateways by connection count first, about 20,000 peak today, redundant across at least two instances.

Q6. What happens when the fan-out bus is down?

Sends still commit and return a sequence, because the response does not wait for the bus. Online clients may not see a push until they sync. A client that syncs on a timer or on reconnect stays correct. I alert on consumer lag. I do not fail the classroom transcript because a push channel is red. The notification nudge is the same pattern: asynchronous, allowed to drop, repaired by the next app open.

Q7. How do read receipts work without a row per reader per message?

Each member stores last_read_seq, updated with a monotonic max. Other clients learn it from a small event and from the summary API. In a cohort, per-message receipts would be members times messages and would still miss offline devices. The UI says "read up to here," which matches the data. If product wants "these three people have read this message" in a direct thread of two, the watermark still answers that. I would add per-message rows only for a scenario that names a small group and a real need.

Q8. Is presence strongly consistent?

No. Presence is a TTL key refreshed by the gateway, allowed to be wrong for about a minute after a crash or a radio drop. It is a hint on the avatar. Membership and the log are the authorities. I do not store presence transitions. I do not route durability through presence; offline users still have the message in the log.

Q9. How would you shard this?

Not at the current 58 peak messages per second and about 180 GB over two years. If we outgrow one primary, the shard key is conversation_id so the seq transaction stays on one node. Member-to-conversation lookups for the home screen would use the member index, which might live as a replicated map or be duplicated onto the user's shard if that query becomes hot. I would not shard by sender. Lesson 30's cost is justified by a measured ceiling, not by the primer's diagram.

Q10. Where does end-to-end encryption fit?

It does not fit this iteration. Cohort channels assume instructors and audited support can read the transcript. Encryption that hides content from the server removes server-side abuse scanning, support, and simple sync, and it requires a key directory this design does not have. I would open a new record that redesigns support access and key distribution together. I would not bolt a "encrypt the body column with a server key" flag and call it end-to-end. That is encryption at rest, which we can do with disk and column encryption without changing the product promise.

Q11. A cohort has 40 online members and the instructor sends one message. Walk the writes.

One transaction updates last_seq and inserts one message row. One bus event is published. Each gateway that has at least one of those sockets writes a frame to each local socket. Presence keys are untouched. Watermarks are untouched until someone reads. Notification hints may enqueue for members with no socket, asynchronously. The write amplification that matters is one row, not forty.

Q12. How do you test this without a thousand browsers?

One test allocates two sequences concurrently and reads them back in order. One test retries the same client id after a commit that "lost" its response. One test runs sync against a primary while the bus consumer is stopped and expects the row. One integration test opens two sockets, kills one gateway, reconnects, and expects a single bubble. A synthetic probe sends a message on a timer and asserts the second socket receives the seq. I do not treat a green unit test of the JSON encoder as coverage of the durability scenario.

Hands-On Exercises

Exercise: Direct-only support desk

Problem statement. TechLearningPro support wants a direct-message desk: a learner opens one thread with the support queue, any on-duty agent may reply, and the transcript must survive agent shift changes. Cohort channels are out of scope for this exercise. Produce a packet that reuses Cohort Wire's log where it fits and says what changes when many agents share one side of a direct conversation.

Tasks.

1. List behavior and out-of-scope items, including whether the learner sees individual agent names or a single "support" sender.
2. Write scenarios for durability, order, and agent handover. Include a numeric window where it matters.
3. Estimate messages per day, concurrent learner sockets, and storage with labeled assumptions. Show peak messages per second.
4. Specify send, sync, and watermark APIs, including idempotency and the status when the learner is not entitled to a thread.
5. Draw the container diagram and the send sequence. Say whether each agent holds a socket and how fan-out stays correct when three agents watch one thread.
6. Decide seq authority and the shard-or-not sentence.
7. Write failure notes for primary, gateway, and bus. State what the learner's client does after a timeout.
8. Record one ADR on "one transcript, many agent sockets" and the option you rejected.

Expected deliverables. Assumptions and arithmetic, API, schema changes, two diagrams, failure notes, ADR.

Evaluation rubric.

A design that stores the transcript only in the agent's browser session, or that promises exactly-once delivery to every device, is not ready.

Lesson Summary

Cohort Wire is an ordered log per conversation with a replaceable push path. Send commits a sequence in one database transaction and returns that sequence. Clients retry with the same client message id. Sync by sequence is how reconnect, a second device, and a dead gateway become correct. The bus and the WebSocket fleet exist so online members see the line quickly. They are allowed to drop frames. Read state is a monotonic watermark. Presence is a TTL hint.

At about 6 messages per second average and about 60 at peak, with about 20,000 sockets and under 200 GB of history for two years, one primary and a pair of gateways are the design. The seq hotspot, the gateway connection count, and a bus outage are the things to operate. Inbox copies, end-to-end encryption, and device-level delivery acknowledgements are different products with different costs, and they stay outside the record until a scenario asks for them.

Lesson 49's shortener could answer from a cache and be slightly stale. A message log cannot answer from a cache that might skip a sequence. That contrast is the point of putting these two cases next to each other. Lesson 51 moves to a system where the bytes, not the order of short texts, dominate the estimate.

Further Reading

• RFC 6455, The WebSocket Protocol — upgrade, frames, and close behavior the gateway relies on.
• Lesson 23, high-level system design methodology; lesson 26, load balancing; lesson 31, consistency; lesson 33, queues and the outbox; lesson 36, coordination; lesson 37, idempotency.
• Spring Framework reference, WebSocket support — TextWebSocketHandler and, if you adopt it, the STOMP broker relay as a bus implementation rather than as the log.
• C4 model.
• Primers: /learn/system-design/whatsapp-messenger, /learn/system-design/mock-interview-chat-system.
• Next case study: design a video streaming platform.