Load Balancing and Traffic Management

Advanced System Design & Software Architecture · Lesson 26 · High-Level Design and Distributed Systems

Lesson Overview

TechLearningPro's classroom fleet had eight healthy processes and one load balancer that did not know the word healthy. A deploy restarted nodes in place. The balancer kept sending learners to a process that had closed its socket, then to the next, and the player retried. A week later a different incident looked like the opposite problem: sessions were sticky, so a single node that had survived a deploy kept a third of the cohort while its neighbors sat idle. Both incidents are traffic management. Adding instances, which lesson 25 refused to treat as a universal fix, does nothing until something in front of them chooses a live target and can take a target out.

A load balancer spreads work across upstreams and, more importantly, stops spreading work to upstreams that should not receive it. This lesson covers layer 4 and layer 7, the algorithms that pick a target, active and passive health checks, readiness versus liveness, sticky sessions and the load they skew, connection draining, and the split between an edge cache for bytes and a balancer for API requests. Global traffic — DNS steering and failover — is here because TechLearningPro Academies already has a home region for progress writes. A second region that receives writes by accident is a data incident wearing a networking costume.

/learn/high-level-design/load-balancer, /learn/high-level-design/cdn, and /learn/system-design/load-balancing introduce the boxes. This lesson is the failure mode and the record: what the balancer is allowed to know, how a node leaves rotation without dropping a watch, and why stickiness is a tax on lesson 25's pool math. The worked path is the classroom API. Video bytes stay on the edge path from lesson 24.

Learning Objectives

By the end of this lesson you will be able to:

1. Place a layer-4 balancer and a layer-7 balancer by what each is allowed to see, and name a decision that requires the higher layer.
2. Choose among round robin, least connections, and a consistent hash, and state the skew each one creates.
3. Separate liveness from readiness, and active checks from passive ones, including what open-source nginx does not check by itself.
4. Explain sticky sessions as a scalability and deploy tax, and say when a token or a shared session store removes the need.
5. Drain a node: refuse new work, keep accepting health failure, finish or bound in-flight requests, then stop.
6. Split CDN edge traffic from API traffic, and state what a global failover may not do with home-region writes.
7. Wire a Spring Boot readiness state to that drain, using the actuator probe endpoints the platform already documents.

Prerequisites

• Lesson 25: horizontal copies help only when the shared budget grows with them, and the API is stateless if you want the balancer to treat nodes as interchangeable.
• Lesson 24's path diagram: video bytes go to the edge, heartbeats and catalog reads go to the API. This lesson balances the second path and caches the first.
• Lesson 2's playback scenario: progress writes stay in the home region. That constraint returns here as a traffic rule.
• Familiarity with HTTP status codes and with a Kubernetes Service or a cloud load balancer at the level of "targets and health." You do not need a vendor certification.
• Spring Boot 3 actuator. The drain uses AvailabilityChangeEvent, ReadinessState, and ApplicationAvailability, which are real types in org.springframework.boot.availability.

Detailed Core Concepts

What the balancer is for

Clients need one address. Processes come and go. The balancer owns the address and a set of targets, and it implements a policy for picking a target and for removing one. That policy is the design. "We have a load balancer" is not a policy. The policy answers:

• Which targets are eligible right now?
• Among eligible targets, which one gets this connection or this request?
• What happens to in-flight work when a target becomes ineligible?
• What does the client observe when none are eligible?
• Which failures should eject a target, and which should not?

The last question is where health checks go wrong. A process can accept TCP, return HTTP 200 on a shallow handler, and be unable to reach its database. A process can be mid-deploy, still TCP-alive, and already guaranteed to die. A process can be busy, which is not the same as dead. Ejecting a busy node can stampede its neighbors. Leaving a dead node in the set turns one crash into a fraction of all traffic. The policy has to say which of those is the ejection condition.

Layer 4 and layer 7

Layer 4, in the language of these systems, balances on the connection: IP addresses and ports, sometimes the TCP payload only as opaque bytes. A layer-4 balancer can forward a TLS connection without terminating it. It cannot route /p/ one way and /api/catalog another way, because it does not read the HTTP path. It can still spread connections, and it can pass the client's address through to the target, which matters for anything that still keys off the source IP. AWS's Network Load Balancer is this class of device. It is fast and thin, and it is the wrong tool for a path-based rule.

Layer 7 balances on the request. It terminates TLS or it at least parses HTTP. It can route by host and path, apply a header, and choose a target per request rather than per connection. HTTP/2 and keep-alive make that distinction sharp: one connection carries many requests, and a layer-4 choice pins all of them to the node chosen at connect time. A layer-7 balancer can send the next request elsewhere. AWS's Application Load Balancer is this class. So is a reverse proxy such as nginx or HAProxy sitting inside the cluster.

The cost of layer 7 is that the balancer is now a HTTP participant. It has a certificate, a request-size limit, a header allowlist, and a failure mode of its own. The benefit is every decision in this lesson that depends on a URL or a cookie. Classroom APIs want layer 7 in front of the Spring Boot nodes: path routing, a readiness URL, and the ability to shed a single request. Video bytes do not want that proxy. They want the edge cache from the capacity envelope, aimed at object storage. Putting both through one layer-7 fleet is how a video miss storm fills the proxy that was supposed to serve heartbeats.

A reverse proxy and a load balancer overlap. The useful distinction is duty, not product. If the process's job is to select among upstreams and to stop selecting a bad one, call it a balancer and hold it to this lesson. If its job is to terminate TLS, compress, and add headers in front of one upstream, it is still on the path and still a capacity element. Many deployments are both, in one nginx or one ALB. Name both duties so a review does not assume the proxy is free.

Algorithms and the skew they create

Round robin walks the eligible list. It is fair when each request costs about the same and connections are short. It is unfair when some requests hold a connection for a long watch heartbeat stream and others are 5 ms catalog reads, or when one node is slower. Round robin does not look.

Least connections sends the next connection to the target with the fewest open connections. It tracks a real imbalance when request durations vary. It can be wrong when connections are a poor proxy for load: a node with few connections can be CPU-bound on a PDF you should not have sent it, and a node with many idle keep-alives can look busy. Use it for the classroom API, where duration varies, and still cap concurrency in the application as lesson 25 did. The balancer's view of "least" is not the primary's view of "pool full."

Weighted variants of either algorithm put more traffic on larger nodes. Weights are how a vertical scale-up joins a fleet without a flag day. A wrong weight is a silent skew. Review the weights when the instance types change.

Power of two choices picks two eligible targets at random and takes the less loaded. It is a practical way to get most of least-connections without a perfectly global counter. You do not need to implement it to know why it exists: full least-connections on a huge list is itself a coordination cost.

Consistent hashing sends the same key to the same target until the set changes, and a set change moves only a fraction of keys. It is the right idea for a cache tier whose hit ratio depends on affinity. It is the wrong default for an API tier you want to drain freely. A hash of the user id that always lands on one JVM is stickiness with extra steps, and lesson 25's pool plan assumed interchangeable nodes.

Source-IP hashing (ip_hash in nginx, or a hash of $remote_addr) pins a client to a node. Carriers and campus NATs put thousands of learners behind one address. The "hash" then pins a cohort to one process. It feels like session affinity and behaves like an outage of everyone else.

No algorithm repairs a saturated primary. Lesson 25 still applies behind the balancer. The algorithm only distributes the work the nodes are actually able to do.

Health: liveness, readiness, active, passive

Liveness asks whether the process should be restarted. A deadlock or a wedged event loop is a liveness failure. Restarting a process that is merely waiting on a slow database restarts all of them and makes the database slower. Liveness is a sharp tool. The check behind it should be narrow: the JVM can still run a local handler.

Readiness asks whether the process should receive traffic. A node that is starting, draining, or unable to reach a required dependency is not ready. It can be alive. Kubernetes will restart on a failed liveness probe and will only remove endpoints on a failed readiness probe. Mixing those is how a database blip becomes a rolling restart of the fleet.

Active checks are the balancer's own requests: a GET on a timer. Passive checks watch real traffic: too many errors or timeouts, and the target is ejected for a cool-down. You want both, aimed at different failures. An active readiness check catches a node that is empty of traffic and already broken, which passive checks cannot see because they need requests. Passive checks catch a node that returns 200 on /ready and 500 on the outline, if your ready URL was too shallow.

Open-source nginx upstreams do passive ejection with max_fails and fail_timeout. They do not include the active health_check directive; that belongs to nginx's commercial build. Pretending otherwise produces a config review that "passes" on a directive the binary will reject, or a fleet that only notices a dead node after learners have failed. HAProxy's option httpchk is an active check and is in the open-source proxy. A cloud load balancer's target-group health check is an active check you configure with a path and a success code. Pick the mechanism your actual binary has, and write the path it will call.

The path for the classroom API is Spring Boot's readiness probe, not a hand-rolled return 200. With probes enabled, /actuator/health/liveness and /actuator/health/readiness are separate. Readiness includes the readiness state and, in our record, the database health indicator. Liveness includes the liveness state only. The balancer polls readiness. The kubelet, if you run on Kubernetes, uses liveness to decide on a restart. Pointing both at the same URL that pings the database is the rolling-restart bug.

Sticky sessions

A sticky session sends a client back to the node that saw it first, via a cookie the balancer inserts or via the source IP. The node may then keep the session in memory. The benefit is real when the session is large and local. The taxes are also real:

• Skew. A cohort that arrived together sticks together. One node runs hot. Lesson 25's "N interchangeable instances" is false, and the pool budget, which divided connections evenly, is a lie on the hot node.
• Deploys. Draining a node drops or migrates everyone stuck to it. If the session is only in that heap, the migrate fails and the learner signs in again, or loses a cart.
• Failure. The node dies and the sticky cookie points at nothing. The balancer must be willing to reassign. A cookie that outlives the node, with no reassignment, is an outage for that subset.
• Scale-in. You cannot remove the quiet nodes if the busy one holds the sticky majority. Autoscaling becomes stuck.

The classroom API does not need stickiness. Authentication is a token the lesson page already sends. Progress is a row in the home-region primary, not a heap object. The balancer should create no affinity cookie. If a future feature needs server-side session state, the state goes in a store every node can read, and the ADR says what that store's outage does to sign-in. Stickiness is the option we refuse in ADR-105, with a reopen condition of a measured session object that cannot move.

Draining

Taking a target out of rotation is a sequence, not a kill.

1. Mark the node not ready. The balancer's next active check, or the endpoint controller, stops sending new work. In-flight requests continue.
2. Wait longer than the check interval plus the deregistration delay. Cloud balancers often keep a target in rotation until a deregistration delay elapses; the AWS ALB default is 300 seconds, which is long enough to matter and too long to discover during the incident. Set it to a value you can explain, on the order of the longest request you still want to finish.
3. The process keeps running during that wait. A preStop hook that sleeps, and an application that has already flipped to refusing traffic, covers the race where the process dies before the endpoint list updates.
4. Bound the wait. A request longer than the drain window is either finished early or abandoned on purpose. WebSocket or long-poll watches need a product rule: the player reconnects to another node. That reconnect is safe because the watch was not pinned.
5. Then stop the process.

Skipping step 1 and sending SIGTERM first drops every in-flight heartbeat. Skipping the wait drops every request that was routed before the control plane learned. Making liveness fail during drain causes a restart loop that fights the deploy. Drain is a readiness concern.

The edge is not the API balancer

Lesson 24 put about 42 Gb/s of video at an edge cache in front of object storage. That cache is a balancer of a sort: it chooses a nearby machine that might already have the bytes. It is not the machine that should run readiness checks against the catalog primary. Public lesson media can be cached at the edge with a long lifetime when the URL is immutable (a content hash in the path). API responses that depend on the caller stay off that cache. Mixing them — a CDN in front of /api/** with a default cache policy — is how one learner's outline is served to another. The traffic split is a security control and a capacity control. ADR-103 already forbade video through the API. This lesson forbids the API through the video cache.

Global traffic and the home region

DNS can send a resolver to a regional address: latency-based answers, or a failover answer when health checks fail. Anycast can attract packets to a nearby site for the edge. Neither mechanism knows that a progress write must land in the home region. If both regions accept writes, you have two primaries and a conflict lesson 29 and lesson 31 will make painful.

The policy for Academies is the one lesson 2 sketched for checkpoints. Media and other immutable bytes may be served from the nearest edge. Catalog reads that tolerate the publish window may be served from a regional read path. Progress writes and enrollment writes go to the home region only. A DNS failover that flips the API's write address to another region is a different, explicit decision, taken when the home region is actually down and a person or a rehearsed automation accepts the data loss or the promotion procedure. It is not the default health-check action. A health check that fails because of a bad deploy in one zone must not promote another continent's database.

Architecture and Design Diagrams

Two doors, because two kinds of bytes were estimated separately.

mermaidCopy
flowchart LR
    learner[Learner]
    dns[DNS]
    edge[Edge cache]
    origin[(Object storage)]
    lb[Layer-7 API balancer]
    api[Classroom API nodes]
    db[(Home-region primary)]

    learner --> dns
    dns --> edge
    dns --> lb
    edge --> origin
    lb --> api
    api --> db

Drain is a sequence. New work stops before the process stops. In-flight work has a bound.

mermaidCopy
sequenceDiagram
    participant O as Operator or deploy
    participant A as API node
    participant B as Balancer
    participant C as Client

    O->>A: mark readiness REFUSING_TRAFFIC
    B->>A: GET /actuator/health/readiness
    A-->>B: not ready
    Note over B: no new requests to this node
    C->>A: in-flight request already accepted
    A-->>C: response
    O->>A: wait out deregistration delay
    O->>A: stop process

A failed readiness check removes traffic. A failed liveness check is allowed to restart. They are different probes.

mermaidCopy
flowchart TD
    probe{Which probe failed?}
    probe -->|readiness| remove[Remove from balancer endpoints]
    probe -->|liveness| restart[Restart the process]
    remove --> keep[Process keeps running]
    restart --> careful[Do not use this for a dependency outage]

Technical Explanation

What the classroom balancer decides

The API balancer is layer 7. It terminates TLS at the edge of the cluster, routes /api/** and /p/** to the classroom target group, and does not route video paths at all. The algorithm is least connections among targets whose last readiness check returned success. There is no stickiness cookie. Passive ejection is a backstop: a target that resets connections is removed even between active checks. The active check interval is 5 seconds, with two failures required, so a single blip does not flap a node, and a dead node is gone within about 10 seconds plus the deregistration delay.

The success condition is HTTP 200 on /actuator/health/readiness. A 503 means not ready. The check does not follow redirects and does not accept a 200 from /actuator/health, because the aggregate health URL can be configured to include components we do not want in the traffic decision, and because liveness and readiness must stay distinct.

When no target is ready, the balancer returns 503 to the client. It does not hold the request in hopes that a node returns. Holding is how a drain of the whole fleet becomes a thread pile on the proxy. The player's retry is the client's decision, and lesson 25 already forbade retrying a timeout into a saturated tier. A fast 503 is easier to retry once, later, than a 30-second hang.

Spring Boot's half of the contract

The process tells the truth on the readiness endpoint. Enabling probes is configuration, documented by Spring Boot:

yamlCopy
management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        readiness:
          include: readinessState,db
  endpoints:
    web:
      exposure:
        include: health

readinessState tracks ReadinessState. db is the existing database health indicator. If the primary is unreachable, readiness fails and the balancer stops sending work. Liveness is left on the default liveness state so a database outage does not restart every pod. Confirm the liveness group does not include db. The default probe groups, once probes are enabled, already separate them; the explicit readiness include is there so a later "helpful" change that adds every indicator to every group fails review.

Drain publishes a state change. AvailabilityChangeEvent.publish is the supported call. The endpoint then fails readiness while the process is otherwise up, which is what step 1 of the drain sequence needs.

javaCopy
package com.techlearningpro.classroom.traffic;

import org.springframework.boot.availability.AvailabilityChangeEvent;
import org.springframework.boot.availability.ReadinessState;
import org.springframework.context.ApplicationContext;
import org.springframework.stereotype.Component;

@Component
public class TrafficDrain {

    private final ApplicationContext context;

    public TrafficDrain(ApplicationContext context) {
        this.context = context;
    }

    public void refuseNewWork() {
        AvailabilityChangeEvent.publish(context, ReadinessState.REFUSING_TRAFFIC);
    }

    public void acceptWork() {
        AvailabilityChangeEvent.publish(context, ReadinessState.ACCEPTING_TRAFFIC);
    }
}

Call refuseNewWork from the deploy hook before the platform removes the instance, and from a preStop command if the only signal you get is the orchestrator. Do not tie it to liveness. A node that is refusing traffic is healthy enough to finish in-flight requests; it is not healthy enough to receive new ones.

Kubernetes, when that is the orchestrator, needs the probe and the sleep that covers endpoint propagation:

yamlCopy
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 20"]

The sleep is not the drain by itself. Without refuseNewWork, a slow readiness interval still admits new requests during the sleep. The application flips first; the sleep covers the balancer learning. terminationGracePeriodSeconds must exceed the sleep plus the longest request you intend to finish. A default grace period shorter than preStop kills the process mid-sleep and looks like a mysterious reset.

If you are not on Kubernetes, the same sequence holds: flip readiness, wait out the load balancer's deregistration delay, stop. The AWS ALB setting is the target group's deregistration delay. Set it to the same bound you put in the grace period, and do not leave the 300-second default unexplained.

A proxy config that matches the policy

HAProxy can express least-connections and an active HTTP check without a commercial add-on. This is the policy, not a full production file: timeouts, TLS, and ACLs still need the operator's standard.

textCopy
backend classroom_api
    balance leastconn
    option httpchk GET /actuator/health/readiness
    http-check expect status 200
    default-server inter 5s fall 2 rise 2
    server a1 10.0.1.11:8080 check
    server a2 10.0.1.12:8080 check

There is no cookie directive. Adding one would implement the stickiness ADR-105 rejects. fall 2 matches "two failures." Nodes that fail the check stay in the file and receive nothing until rise 2 clears them, so a restarting node does not take a request on its first accidental success.

For an open-source nginx upstream, do not paste an active health_check block and assume it runs. Use passive max_fails and fail_timeout as a backstop, and put the active readiness poll on a balancer that actually supports it, or on the orchestrator probes above. One active checker is enough. Two active checkers with different paths are how a node is "up" and "down" in two dashboards.

Global rule, written so DNS cannot overrule it

Public media hostnames resolve to the edge. The API hostname resolves to the home region's layer-7 balancer. A second region may hold a warm API fleet for a rehearsed failover. Its balancer is not in the ordinary DNS answer. Failover is a record change plus a database promotion decision, owned by the recovery procedure, not by a five-second health check. Zone failure inside the home region is handled by targets in more than one zone behind the same regional balancer, which does not change the primary's region.

That split prevents the failure mode where a zone health check, too aggressive, swings writes to a replica that was never promoted and is still applying lag. Learners would see progress move backwards. The traffic layer does not get to cause that.

Practical Implementation or Design Walkthrough

Problem statement

Spread classroom API traffic across a stateless fleet, remove nodes that cannot serve, drain a deploy without dropping every in-flight heartbeat, keep video off this balancer, and keep progress writes in the home region.

Policy

Failure notes

Node process crash. Active check fails, target leaves, other nodes take the next request. In-flight work on the dead node is lost. Clients retry a heartbeat; the upsert is idempotent on learner and lesson, so a replayed position write converges. A non-idempotent create still needs the idempotency key from lesson 23.

Primary down. Readiness includes db, so every API node goes not-ready. The balancer returns 503. That is an outage of the API, which is honest. Restarting the nodes via liveness would add a thundering herd when the primary returns. This is why db is not on the liveness probe.

One zone lost. Targets in the surviving zone stay ready. Capacity falls by the lost zone's weight. Lesson 25's connection budget was computed for the full fleet; the surviving zone must still fit on the primary, which it will if the budget was estate-wide. The risk is the autoscaler replacing lost nodes so fast that pools overshoot. Cap the surge.

Deploy. New nodes become ready only after the application has accepted traffic and, if you require it, after a warm-up. Old nodes flip to refusing, drain, and stop. Least connections moves work without a sticky residue. A sticky cookie would have piled the cohort on the last old node.

Bad configuration points the CDN at the API. Video range requests arrive at Tomcat. The balancer is "up" and the site is down, because the pool is full of byte streams. The mitigation is a separate hostname and a balancer rule that does not list video paths. Detection is request size and path on the API access log. This failure is a routing failure, not a scale failure.

Decision record

ADR-105: API nodes are interchangeable; the edge is a different door; writes stay in the home region

• Status: Accepted
• Context: Deploys dropped in-flight calls, and a sticky cohort melted one node. Video egress is the lesson 24 envelope. Progress writes are home-region by the lesson 2 scenario.
• Options: (A) Source-IP stickiness and one fleet for video and API. (B) Layer-7 least-connections, readiness-based membership, explicit drain, no affinity, video on the edge hostname, DNS failover of writes only as a rehearsed promotion. (C) Global anycast of the write API.
• Decision: Option B. Deregistration bound is 30 seconds, aligned with the grace period. Readiness includes readinessState and db. Liveness does not include db.
• Consequences: A server-side session in the heap will not work, and we will not add a cookie to make it work. Cross-region write failover is slower and requires a person or a named automation. We accept that over a health check that promotes a replica by accident. The 30-second drain means a request longer than 30 seconds can be cut; the player reconnects.
• Reopen when: a feature requires a session object that cannot move, with a measured size, or when a rehearsed regional promotion becomes the standard response to a zone loss. Zone loss alone does not reopen it.

Validation

• Kill a node's network and confirm it leaves the target set within two check intervals, and that liveness did not restart its neighbors.
• Start a deploy and confirm new requests stop arriving at the draining node while an in-flight request that began earlier still completes.
• Point a client at a signed-in notes call and confirm the balancer set no affinity cookie.
• Resolve the media hostname and the API hostname and confirm they differ.
• Fail the primary in a test environment and confirm readiness goes red and pods are not crash-looping.
• Confirm a request to the API for a video path is not how bytes are served in production. That last test is a routing test.

Expected result

A deploy is a readiness change plus a wait, not a packet loss event. A hot node is a capacity problem you can see in least-connections counts, not a sticky accident. A regional DNS idea has to survive the sentence "this would give us two writers."

Enterprise Real-World Scenarios

The cohort that stuck to one node

A Friday release turned on a session cookie "temporarily" so an experiment could keep a learner on the node that had cached their outline in memory. The cache was a HashMap. The balancer was configured with a cookie insert because that was easier than a shared cache. By evening one node held the largest cohort. CPU there was saturated. The other seven nodes were idle. Autoscaling added an eighth idle node, because the average CPU of the fleet was fine. Lesson 25's pool plan assumed even spread. The hot node opened its full pool and waited on the primary, while the estate-wide connection count still looked legal.

The repair was to delete the cookie and the map. Outline caching, when it returned, was a shared cache with an explicit TTL, which is lesson 27, not a heap. The incident write-up banned affinity cookies on the classroom API unless ADR-105 is superseded. The metric added to the deploy checklist is the ratio of busiest node request rate to the mean. Least-connections without stickiness keeps that ratio near 1 for similar requests. A ratio of three is a page, not a curiosity.

The same release dropped heartbeats during rollout because the old process was stopped before readiness flipped. Learners saw the player retry a dozen times. The drain sequence, including the preStop wait, shipped the following week. Both bugs were traffic policy. Neither was fixed by a larger instance.

A zone health check that almost promoted a continent

Academies runs the progress primary in one region. A proposal from a platform template enabled DNS failover: if the regional balancer's health checks failed three times, the API name would flip to a second region whose database was a replica. The health check was a TCP connect to the balancer. During a bad deploy the home balancer answered TCP and failed readiness on every target. The check, being TCP, stayed green, so DNS did nothing. That accident of a shallow check was the only reason writes did not split.

The review treated the near miss as a design bug, not as a lucky dashboard. ADR-105 says a TCP connect is not the failover signal, and a readiness failure of the fleet is not a promotion of another region's replica. Zone loss is absorbed by targets in the other zone of the same region. Region loss is a disaster-recovery decision with a named operator, a promotion step, and a rehearsal. The template was changed so failover records are opt-in per hostname. The media hostname may fail over to another edge. The API write hostname may not.

The deeper point is the one lesson 2 made about defining "down." The template defined down as "TCP failed." Learners define down as "progress did not save" or "progress saved in two places and then jumped backward." Traffic management that uses the wrong definition will automate the wrong recovery. The record is what stops the template from returning in the next account.

Best Practices

• Decide layer 4 or layer 7 from the facts the policy needs. Path, cookie, and per-request readiness are layer 7. Opaque TCP pass-through is layer 4.
• Keep video on an edge hostname and the API on a balancer that has never seen a byte-range stream.
• Use least connections when request time varies, and still enforce the application pool cap. The balancer does not know the primary is full.
• Avoid source-IP affinity. NATs make it a herd.
• Poll readiness, not aggregate health, and do not put the database on the liveness probe.
• Use an active check your actual proxy supports. Do not depend on an nginx directive the open-source build ignores.
• Drain by flipping readiness, waiting out the deregistration delay you set on purpose, then stopping. Align grace periods with that wait.
• Return 503 when the target set is empty. Do not build an unbounded queue on the proxy.
• Treat stickiness as a design you must pay for in skew, deploys, and scale-in. Prefer a token and a shared store.
• Keep global DNS failover of write paths out of ordinary health checks. Promote a database only with the recovery procedure.
• Watch the busiest-to-mean request ratio. It catches stickiness and broken weights.
• After a deploy, confirm in-flight requests completed on the draining node and that new requests did not arrive.

Common Mistakes and Anti-Patterns

TCP as the definition of healthy. The balancer connects and calls the node alive while the primary is unreachable or the process is still starting. Learners receive errors the check cannot see. The repair is an HTTP readiness path that includes the dependency the request cannot live without, and a liveness path that does not.

Liveness that checks the database. A blip restarts the fleet, which reconnects together and deepens the blip. The repair is the probe split in this lesson. Readiness removes traffic. Liveness restarts a wedged process.

Stickiness as a cache. A local HashMap plus a balancer cookie avoids a shared cache and creates a hot node the autoscaler cannot see in the average. The repair is a shared cache or no cache, and ADR-105's ban on the cookie.

Drain by kill. SIGTERM and a short grace period race the endpoint update. In-flight work dies, clients retry, the remaining nodes take the spike. The repair is the sequence: refuse, wait, stop, with the application participating via ReadinessState.

One fleet for bytes and API calls. The layer-7 proxy or the API JVM starts carrying video because the hostname was convenient. A launch of a popular lesson fills the threads lesson 25 capped for outlines. The repair is separate hostnames and an envelope that makes the byte path obvious.

Health-check failover of a replica. DNS flips writers because a check failed, and the other region is not promoted, or is promoted without a fence on the old primary. The repair is to keep write DNS in the home region and to make promotion a rehearsed step. Shallow checks failing open is not a conservative default; it is an unreviewed promotion.

Two active checkers, two truths. The orchestrator polls /ready and the proxy polls /health, and a node flaps in one and not the other. Operators follow the green one. The repair is one readiness URL, referenced by every checker that is allowed to remove traffic.

Troubleshooting and Design Review

Symptom: During deploys, error rate spikes and then settles, and players retry heartbeats in bursts.

Likely cause: targets are removed by killing the process, or the deregistration delay is shorter than the time the control plane needs, or preStop exceeds the grace period and the process is killed during the sleep.

Investigation: timestamps of readiness flip, endpoint removal, and process stop. Compare to the grace period and the deregistration delay.

Corrective action: align those numbers, flip REFUSING_TRAFFIC first, and confirm an in-flight call still finishes in a staging deploy.

Symptom: One node runs hot; the fleet average looks fine; a cookie named like a route id appears on responses.

Likely cause: stickiness, source-IP hash behind a NAT, or weights that do not match instance size.

Investigation: request rate per node, the cookie, and whether ip_hash or an equivalent is set. Check a large cohort's source address.

Corrective action: remove the affinity, move any heap session into a store or delete it, and recheck the busiest-to-mean ratio after the next peak.

Symptom: All API nodes restart together when the primary blips.

Likely cause: liveness and readiness share a check that touches the database.

Investigation: probe configuration and the Spring health groups. Look for db on the liveness include list.

Corrective action: liveness stays local. Readiness includes the database. Re-test by blocking the primary and watching restart counts stay flat while traffic stops.

Symptom: The API tier's network graphs spike when a lesson video is assigned in class.

Likely cause: media URLs point at the API host, or the CDN falls through to the origin via the API.

Investigation: the hostname in the player, the path on the access log, request sizes.

Corrective action: media URLs on the edge host, immutable paths where the bytes are versioned, and no video location in the API target group. This is the ADR-103 path enforced in routing.

Symptom: A regional incident playbook says "DNS will fail over," and nobody has named the database.

Likely cause: a template equated balancer health with region promotion.

Investigation: the record that DNS points at, the replica's role, and whether writes would be accepted.

Corrective action: remove write failover from automatic DNS. Keep multi-zone targets inside the home region. Put region loss in the disaster-recovery rehearsal, not in the five-second check.

Design-review questions: What is the balancer allowed to see, and is that layer 4 or layer 7? Which algorithm, and what skew does it create behind a NAT? What URL defines readiness, and does liveness avoid the database? How long is drain, and which of the application, the grace period, and the deregistration delay is the longest? Is any cookie pinning clients? Where do video bytes go? If the home region disappears, what must a human do before writes are accepted elsewhere?

Interview Questions and Answers

Q1. What is the difference between a layer-4 and a layer-7 load balancer, and when do you pay for layer 7?

A layer-4 balancer forwards connections using addresses and ports and can pass TLS through. A layer-7 balancer understands HTTP, so it can route on host and path, check a readiness URL, and send two requests on one connection to different targets. I pay for layer 7 when the policy needs those facts, which the classroom API does. I do not put video byte streams through that proxy. The cost is a certificate, request limits, and another component that can return 503. AWS NLB is the thin class; an ALB or HAProxy is the HTTP class.

Q2. Why is least-connections a better default than round robin for this API, and what does it still miss?

Request times vary: a cheap health check and a heavier outline are not equal, so round robin overloads the node that drew the long calls. Least connections steers toward the less busy node. It misses the primary's connection budget, a CPU-bound task that holds few connections, and any stickiness you added beside it. I still cap pools in the application. The balancer is not the saturation detector for the database.

Q3. Why is source-IP hashing a poor session strategy for learners on a campus network?

Many learners share one public address. The hash pins that entire population to one target, so a cohort becomes a hot node and the rest of the fleet idles. A balancer cookie at least pins per client, and it still has the deploy and skew taxes. For this API I want neither. The session is a token, and the progress row lives in the database. Affinity is a workaround for state in the heap.

Q4. How should liveness and readiness differ?

Readiness answers "send traffic or not." It may include the database, and a failure only removes the node from the set. Liveness answers "restart or not," and it should stay local to the process. If liveness checks the database, a blip restarts every node and the reconnect storm makes the blip worse. Spring Boot exposes /actuator/health/readiness and /actuator/health/liveness when probes are enabled. I point the balancer at readiness and I keep db out of the liveness group.

Q5. What is wrong with using TCP connect as the health check?

TCP proves the port accepts a handshake. It does not prove the outline query can run, that the process has finished starting, or that it is draining. I have seen a shallow TCP check stay green while every HTTP target was failing readiness, which accidentally prevented a bad DNS failover and also failed to remove broken nodes. The check should be the HTTP readiness URL and a 200, with a defined failure count so one blip does not flap.

Q6. Walk through a drain.

Flip ReadinessState.REFUSING_TRAFFIC so the readiness endpoint fails while the process still runs. Wait until the balancer and the endpoint controller have observed that, which is the deregistration delay plus a check interval, covered by a preStop sleep if the orchestrator only gives you a kill signal. Finish in-flight work up to that bound. Then stop. The grace period must be longer than the sleep. I do not fail liveness to drain, because that invites a restart. Requests longer than the bound, such as a parked connection, must reconnect on another node, which is safe only because we did not pin them.

Q7. What are the costs of sticky sessions?

Skew, because clients clump on the node they first saw; painful deploys, because that node's heap holds state you cannot move; a dead node that still owns a cookie; and scale-in that cannot remove the hot node. Averages hide the skew, so autoscaling adds idle capacity. I accept stickiness only when a measured state cannot live anywhere else, and then I would rather put that state in a shared store and keep the nodes interchangeable. The reopen condition belongs in an ADR, not in a temporary cookie that ships on Friday.

Q8. How do you keep a CDN from caching or carrying the API?

Separate hostnames and paths. Immutable media URLs, with a content hash when the bytes are versioned, can sit at the edge for a long time. Authenticated API responses stay on the layer-7 balancer with a cache policy that does not store them at the edge. A default "cache everything on this host" rule will eventually store one tenant's outline under a URL the next tenant can fetch. The capacity reason matches the security reason: video is tens of gigabits, and it must not occupy the API threads.

Q9. When should DNS failover send traffic to another region?

For the edge, a failed region can be a DNS or anycast change because the bytes are immutable and the origin story is already a miss path. For progress writes, a DNS change is a database promotion. I do not let a five-second health check do that promotion. Multi-zone targets inside the home region cover a zone loss. A region loss is the disaster-recovery procedure, with a fence on the old primary so it cannot accept writes after the new one does. Automating the DNS flip without the fence is how you get two writers.

Q10. The fleet average CPU is low and one node is melting. What do you check first?

Affinity: a cookie, ip_hash, or a consistent hash on user id. Then weights, then a local cache that exists only on the node that happened to serve a popular key. I look at the busiest-to-mean request ratio rather than the average. The repair on the classroom API is to delete the affinity and any heap session. I do not add a ninth node until the eight I have are actually eligible to receive the cohort.

Q11. Open-source nginx is in the design. Where do active health checks come from?

Not from the open-source health_check directive, which is a commercial feature. Open-source upstreams can eject passively with max_fails and fail_timeout, which only reacts after real requests have failed. I put the active readiness poll on HAProxy, on a cloud target-group check, or on the orchestrator, and I point it at /actuator/health/readiness. I do not draw an active check the binary will not run.

Q12. How does this lesson depend on the scalability lesson?

Lesson 25 said extra instances do not help a serial primary, and that pool size must fall as instance count rises. The balancer makes those instances reachable and makes instance count change during a deploy and an autoscale. If I add stickiness, the even pool division is false. If I drain by killing, the retry spike violates the "do not retry into saturation" rule. If I fail readiness on a deep check and also restart on it, I create the reconnect storm lesson 25's knee cannot absorb. Traffic policy is how the horizontal fleet stays inside the assumptions of the capacity and latency records.

Hands-On Exercises

Exercise: Regional traffic policy with failover

Problem statement. TechLearningPro Academies will add a second region for media and a warm standby for the API. Design the traffic policy. Progress and enrollment writes remain home-region unless a rehearsed promotion says otherwise. Use the classroom envelopes you already have; do not invent a new bitrate.

Tasks.

1. Draw the doors: edge, API balancer, primary, standby. Mark which hostnames exist.
2. Choose layer 4 or layer 7 for the API and justify it with one decision you must make per request.
3. Specify readiness, liveness, the check path, and the failure count. State what must not be on the liveness probe.
4. Write the drain sequence with a numeric deregistration bound and the grace period that contains it.
5. Take a position on stickiness and record the skew it would create for a 200-learner cohort behind one NAT.
6. State the automatic behavior when one zone in the home region dies, and the non-automatic behavior when the whole region dies.
7. List the failure mode where video arrives at the API, and the routing rule that prevents it.
8. Write an ADR that rejects automatic DNS promotion of the standby database, with the condition that would reopen it.

Expected deliverables. The diagram, the probe and drain policy, the zone-versus-region note, and the ADR.

Evaluation rubric.

A single health URL for restart and traffic, a sticky cookie as the availability plan, or DNS that promotes a replica on a TCP failure means the policy is not ready for review.

Lesson Summary

A load balancer is a policy for membership and for choice, not a box that makes a fleet scalable. Layer 7 earns its place when the API must route and must observe readiness; video stays at the edge because the byte envelope and the privacy rule both say so. Least connections spreads uneven requests and still cannot see a full database pool. Readiness removes traffic. Liveness restarts a wedged process and must not include the database. Stickiness buys a hot node and a cruel deploy. Draining is a readiness flip, a wait you set on purpose, and then a stop.

Global traffic follows the same honesty. Multi-zone targets cover a zone. A second region receives writes only after a promotion someone rehearsed, not because a health check flipped DNS. TechLearningPro's two incidents — dropped deploys, and a cohort glued to one heap — are what ADR-105 refuses to repeat.

Lesson 27 puts a shared cache on the read path the balancer has just made stateless, and it spends the rest of the lesson on the staleness that cache will introduce.

Further Reading

• Spring Boot reference, "Liveness and readiness," and the actuator health endpoint — ReadinessState, probe groups, and the URLs used here.
• Kubernetes documentation on container probes, preStop, and terminationGracePeriodSeconds — the drain race the sleep is there to cover. Service sessionAffinity: ClientIP is the stickiness option this lesson declines.
• HAProxy configuration reference for balance leastconn and option httpchk — active checks in the open-source proxy. The nginx open-source upstream module documents max_fails and fail_timeout; active health_check is documented as a commercial feature.
• AWS Elastic Load Balancing guides for Network Load Balancers, Application Load Balancers, and target-group deregistration delay — layer 4 versus layer 7, and the default delay you should not discover in production.
• RFC 9110 — 503 as the honest answer when no target is ready.
• Primers: /learn/high-level-design/load-balancer, /learn/high-level-design/reverse-proxy, /learn/high-level-design/cdn, /learn/system-design/load-balancing, /learn/system-design/cdn-and-edge.
• This course: /learn/advanced-system-design/scalability-and-performance-fundamentals, /learn/advanced-system-design/caching-strategies-and-cache-invalidation, /learn/advanced-system-design/disaster-recovery-and-business-continuity.