Scalability and Performance Fundamentals

Advanced System Design & Software Architecture · Lesson 25 · High-Level Design and Distributed Systems

Lesson Overview

TechLearningPro's catalog API met its Thursday budget and missed the same budget on launch morning, after the team had doubled the instance count. CPU on the new instances sat near 30 percent. The primary's connection slots were exhausted, and the application threads spent their time waiting on a pool. The incident report said "we need to scale." The incident was a saturated dependency behind a serial fraction of the request. More application processes made the pool queue longer.

Scalability and performance answer different questions. Performance asks how long one interaction takes, and how many complete per second, on a stated path. Scalability asks whether a stated growth in workload can be absorbed by a stated kind of resource while that performance stays inside its measure. Lesson 2 defined both as quality attributes. Lesson 24 produced the rates. This lesson is the mechanics that connect a rate to a latency: vertical and horizontal scaling, the bottleneck that actually saturates, Amdahl's limit on parallel speedup, a latency budget split across hops, and the queueing fact that latency bends upward as utilization approaches one.

/learn/high-level-design/horizontal-vertical and /learn/system-design/horizontal-vs-vertical-scaling name the two directions. This lesson adds the math a review can recompute, the failure mode where the "scaled" tier was never the constraint, and a decision record that spends the launch budget on the serial hop. You will not find a core-count formula. You will find a way to decide which element is allowed to receive the next unit of capacity.

Learning Objectives

By the end of this lesson you will be able to:

1. State scalability as a multiplier, a resource, and a quality that must still hold, and keep that statement distinct from a latency number.
2. Choose vertical or horizontal growth for a concrete limit, including pools, sessions, and a single primary.
3. Apply Amdahl's law to a request with a serial fraction and explain why a large N stops helping.
4. Split a latency budget across serial hops and parallel hops, and account for fan-out at the tail.
5. Use utilization and Little's law to explain why a "healthy" 90 percent busy resource misses a percentile.
6. Trace a latency incident to the saturated element and write the next capacity move as an ADR.
7. Recognize retry amplification and pool math as scalability bugs, not as mysteries.

Prerequisites

• Lesson 2's latency and scalability scenarios: a percentile, a window, and a clock boundary. This lesson spends those budgets. It does not redefine them.
• Lesson 24's envelope. The launch catalog burst of a few thousand reads per second is the workload this lesson diagnoses. Heartbeat QPS and video egress stay in their own envelopes.
• Comfort with a fraction. Amdahl's formula is one division. The M/M/1 sketch is optional and labeled as a teaching model.
• Spring Boot 3's server and pool settings as configuration, not as a new framework: server.tomcat.threads.max and HikariCP's maximumPoolSize. Defaults cited here are the documented ones: Tomcat's max threads at 200 in Spring Boot, HikariCP's default maximum pool size of 10, PostgreSQL's default max_connections of 100.
• Lesson 26 will place a load balancer in front of horizontal copies. This lesson decides whether those copies can help.

Detailed Core Concepts

Performance is a path; scalability is a growth story

A latency measure names one path. "Catalog outline, p95 under 200 ms at the API, over the launch window" is a performance scenario. Throughput on that path is how many outlines complete per second while the latency measure holds. You can raise throughput by batching and hurt a single learner. You can cut the median with a cache and leave the tail on the miss path. Report the percentile the scenario named. The mean is a diagnostic, and lesson 2 already explained why it is a poor contract.

Scalability is the claim that a multiplier on the workload, absorbed by a named resource, leaves that measure intact. "Ten times catalog reads, by adding stateless API instances, p95 still under 200 ms, until the primary's connection budget" is a scalability claim. It has a boundary built in. A claim with no multiplier is a wish. A claim with no resource is a slogan. A claim that forgets to say which quality holds will be satisfied by a design that "scales" into a timeout.

Vertical scaling adds CPU, memory, or I/O to one node. It preserves a single address space, so in-memory state and a single writer remain simple. It ends at the largest node you will pay for and operate, and it concentrates failure: when that node stops, the path stops. Horizontal scaling adds nodes. It requires a story for state, for sessions, for anything that must happen once, and for the shared dependency every node still calls. The database does not become horizontally scalable because the API did.

The practical order is the one reviews resist. Find the saturated element. Scale that element in the direction its constraint allows. Vertical is often the right first move for a primary whose CPU is the limit and whose working set fits a larger machine. Horizontal is the right move for a stateless API whose CPU is the limit and whose downstream pools have been resized to match. Doing the second while the first is true produces the launch incident: idle CPUs, exhausted connections.

Bottlenecks, utilization, and saturation

A bottleneck is the element whose limit caps the path. Utilization is how busy it is: busy time over available time, or arrival rate over service rate for a single server, ρ = λ / μ. Saturation is the queue forming because utilization is at the cap: threads waiting, connection pools timing out, run queues growing. A resource can be the bottleneck at 40 percent utilization if a lock serializes it. Utilization is a clue. Saturation is the symptom learners feel.

Brendan Gregg's distinction is the one to keep in a review. High utilization with no queue can be fine. Low utilization with a queue means the thing you graphed is not the thing they are waiting on, or the service time is uneven. During the catalog launch the API CPU was a low-utilization graph. The pool was saturated. Dashboards that only show CPU will clear the guilty tier.

Latency bends before the server "falls over"

For a single queue with Poisson arrivals and exponential service (the M/M/1 teaching model), mean response time is service time divided by (1 − ρ). At 50 percent utilization the mean is twice the service time. At 80 percent it is five times. At 90 percent it is ten times. The assumptions are strict: one server, memoryless distributions, a stable arrival rate, no timeouts, no retries. Real request times are more variable, and variability makes the wait worse. Kingman's approximation for a single server says the queue wait grows with (c_a² + c_s²) / 2, the squared coefficients of variation of arrivals and service, times ρ / (1 − ρ), times the service time. You do not need the formula in an interview to remember the shape. You need it so you do not treat 90 percent busy as "10 percent headroom, latency unchanged."

The knee is why lesson 24's headroom exists. A design peak at 70 percent of the measured capacity of the bottleneck has a chance of holding a percentile. A design peak at 95 percent will miss p99 even when the mean still looks polite. This is not permission to buy at 2× without measuring service time. It is permission to distrust a plan whose math only checks out at ρ near 1.

Timeouts and retries move the arrival rate. A client that times out and retries once has, from the server's point of view, raised λ. If the server was slow because ρ was high, the retry increases ρ further. That feedback is a scalability failure mode. A budget that sheds load, which lesson 35 designs, is one response. A budget that does not retry a read that is already late is another. Unbounded retries are not a reliability feature on a saturated path.

Amdahl's law

Amdahl's law bounds the speedup from parallelizing part of a fixed piece of work. If a fraction s of the time is serial and must stay serial, and the rest is parallel across N workers:

speedup(N) = 1 / (s + (1 − s) / N)

As N grows, speedup approaches 1 / s and then stops. The law does not know about networks, caches, or coordination you add when you split the work. Those costs make real speedup worse than the formula. The formula is already enough to kill a bad plan, so treat it as an upper bound.

Apply it to one request before you apply it to a fleet. A catalog outline that spends 80 ms in a single primary query and 40 ms in local rendering has a serial fraction of 80/120, about 0.67, if the query cannot be split. Ten API instances do nothing to that 80 ms. The request's latency floor is near 80 ms no matter how many JVMs render the other 40 ms. Ten instances do raise how many such requests you can run at once, until the primary's 80 ms service, multiplied by concurrency, saturates the primary. Horizontal API scale moves throughput only while the serial hop has spare service rate. It does not move the serial hop's own latency.

Worked numbers for that outline, service times at low load:

The table is the upper bound on one request's compute, assuming the parallel slice splits perfectly and the serial query does not slow down as N grows. In production the query does slow down as N grows, because every instance sends it. The table is therefore optimistic twice. It is still the right slide for a review that wants a fiftieth instance to shave the p95.

There is a second, fleet-level reading. If the workload is many independent requests, and each request is independently served by one node, the serial fraction inside the request still sets that request's floor, while throughput scales with N until a shared serial resource saturates. People quote Amdahl as "you cannot scale horizontally," which overreaches, and they ignore it as "only for CPUs," which underreaches. The shared primary is the serial fraction of the fleet.

Latency budgets

A budget assigns the scenario's milliseconds to hops before anyone writes code. For a catalog outline with a 200 ms p95 at the API process boundary, a review-sized split on a launch day might be:

Serial allowances add. Parallel allowances count once, as the maximum, plus a small join cost. The spare is not a gift to the next feature. Tail latency consumes it. If five independent downstream calls each meet their own p99, the chance they all do on the same request is 0.99^5, about 95 percent, when the tails are independent. The join is a worse percentile than any participant. Budgets that add p99s as if they were p50s pass in design and fail in the launch window. Prefer fewer serial calls over a clever parallel fan-out you did not measure.

The spare also funds garbage collection, a noisy neighbor, and a retry you hope not to make. Spending it in the design meeting means the production percentile has nowhere to land.

State the failure when a hop exceeds its allowance. For the outline, exceeding the primary allowance returns a stale cache if one exists and is allowed, or a 503 if not. It does not wait 30 seconds and ruin the pool. The budget is an architectural control only if the timeout is at or inside the allowance. A client timeout of two seconds and a budget of 200 ms means the budget is a slide.

Pools and the hidden horizontal limit

Each API instance holds a HikariCP pool. The documented default maximum is 10 connections. Fifty instances at the default are 500 connections aimed at PostgreSQL, whose default max_connections is 100. You will not get there: connections will fail, or someone will raise max_connections until the primary spends its memory on backends and its CPU on context switches. The scalable-looking fleet has a serial resource with a hard slot count.

The repair is arithmetic before the launch. Decide the primary's connection budget from a benchmark, not from a hope. Divide that budget by the instance count. Set spring.datasource.hikari.maximum-pool-size to the quotient. If the quotient is 2 and the workload needs more concurrency than two connections per node can provide, you have too many instances or too small a primary. Adding a sixth-tenth instance with the default pool of 10 makes the incident worse.

Tomcat's server.tomcat.threads.max defaults to 200 in Spring Boot. Two hundred threads blocked on a pool of ten is a queue inside the JVM. The threads-running metric looks busy. The CPUs look idle. The learner sees the timeout. Match the threads that block on the database to a number the pool can actually serve, and shed or queue explicitly beyond that. An infinite accept-and-block policy is how utilization graphs lie.

Vertical scaling, with the costs named

A larger primary helps when the limit is CPU, working set in RAM, or I/O on that one machine, and when the operational story of one writer is still the design. It hurts when the limit is a lock, a single session, or a connection slot you will not raise, and it hurts the blast radius. A vertical move that doubles RAM and quiets disk reads is a legitimate scalability action. It is not a moral defeat beside a horizontal diagram.

Garbage collection is the vertical tax people forget. A heap grown to "be safe" can pause longer than the latency budget. The pause is a serial stop of the process. It will not show up in an Amdahl table that only counts the SQL time. If you scale a JVM vertically, you keep the pause inside the spare milliseconds, or you split the work onto more heaps. Measure pauses. Do not assume a bigger machine is a faster request.

Coordinated omission, briefly

Load generators that wait for a response before sending the next request, and then record only the successful samples, under-report the tail. Gil Tene named this coordinated omission: the system was too slow to accept the work, and the missing work is exactly the bad latency. A scalability test that reports a pretty p95 while the client was blocked is measuring a different, lighter workload. This lesson's rule is small: the test's offered rate must be the launch envelope's rate, and timeouts count in the percentile. Lesson 65 builds the full performance review. Here, refuse a graph that could not have observed the queue.

What horizontal copies still share

Stateless application nodes still share the primary, the cache, the identity provider, the deployment, and often the availability zone. A bug in the outline query is on every node the moment you deploy it. Horizontal scale is a throughput tool. It is not isolation. Bulkheads, which lesson 41 covers, are how one noisy call stops stealing the pool. Until then, a separate pool for the launch-critical outline is a legitimate, smaller move than a new service.

Session state in the JVM is a horizontal blocker. A learner pinned to an instance by a session map cannot move when that instance drains, and the fleet's capacity is uneven. Lesson 26 handles stickiness as a load-balancer choice. The performance view is simpler: memory sessions make horizontal scale partly fictional. Put the session in a store you can name, or keep the API stateless and let the session live in the token. The trade-off is token size and revocation latency, which is a security scenario, not a free win.

Architecture and Design Diagrams

One catalog request on launch morning, with the serial hop drawn as its own box so a reviewer cannot "scale" it by coloring the API green.

mermaidCopy
flowchart LR
    client[Learner]
    api[Catalog API instances]
    pool[Connection pool per instance]
    primary[(Catalog primary)]
    cache[(Outline cache)]

    client --> api
    api --> cache
    api --> pool
    pool --> primary

The budget combines in time, not in the org chart. Parallel hops share a window. The primary sits on the critical path.

mermaidCopy
flowchart TD
    start[Clock starts at API] --> auth[Auth check 10 ms]
    auth --> outline[Outline 80 ms]
    outline --> join{Parallel window 60 ms}
    join --> badge[Enrollment badge]
    join --> notes[Notes snippet]
    badge --> assemble[Assemble 15 ms]
    notes --> assemble
    assemble --> spare[Spare 35 ms unassigned]
    spare --> done[p95 target 200 ms]

Amdahl's ceiling for the 80/40 split is the curve the fleet cannot beat on a single request:

mermaidCopy
flowchart LR
    n1[N = 1: 120 ms] --> n2[N = 2: 100 ms]
    n2 --> n10[N = 10: 84 ms]
    n10 --> nInf[N large: 80 ms floor]

Technical Explanation

Where the launch actually waited

Lesson 24's launch envelope is about 2,700 catalog reads per second at the burst, against a Thursday habit of about 140. Suppose a single outline query takes 8 ms of primary time when the primary is idle, and the API spends 4 ms of CPU. One primary at 8 ms can theoretically complete 125 such queries per second if it does nothing else and ρ is allowed to reach 1, which it must not. At ρ of 0.7 the teaching model allows about 90 queries per second before the queue dominates. That crude service-rate sketch is not a benchmark. It is enough to show that 2,700 QPS will not fit on the serial hop.

A cache that absorbs 95 percent of outline reads leaves 5 percent, about 135 QPS, for the primary during the burst. That is in the neighborhood of the sketch, still tight, and only true if the cache is warm and the key is stable. The scalability move is the cache plus a connection budget, not a fortieth API instance. If the cache cannot be allowed to serve an outline — a freshness scenario that says zero — the primary must be scaled vertically or the read model must be split, and the ADR has to say which.

Enrollment-badge calls that fan out to a second service add a parallel hop. They do not remove the outline from the critical path. Making them synchronous and serial would add their allowance to the 80 ms and shrink the spare to nothing. Leave them parallel, give them a timeout inside the 60 ms window, and define the outline response when the badge is late: omit the badge. That degraded response is a product decision recorded with the budget.

Retry math on the same burst

If 10 percent of calls exceed the client timeout and each is retried once, the primary sees 1.1 times the offered load, and the slow calls are exactly the ones being retried. At ρ already high, Kingman's ρ / (1 − ρ) term is steep: moving from 0.8 to 0.88 is not a 10 percent latency change. It is the difference between a factor of 5 and a factor of about 8 on the service time, in the M/M/1 sketch, before variability. The policy used here: catalog GETs are retried at the client only on a connection failure, never on a timeout that already consumed the budget. The idempotent read is safe to retry in terms of correctness and unsafe to retry in terms of saturation.

Pools for a fleet of twenty

A benchmark, imagined as the one ADR-104 will require, says the primary stays inside 30 ms p95 at 200 concurrent queries and falls over by 400. Set the estate's total pool near 200, not near 400. Twenty API instances receive a maximum pool of 10 each, which happens to be Hikari's default, so the default is safe only at that instance count. The fortieth instance requires the pool size to drop to 5, or the primary's budget to be re-benchmarked after a vertical move. Writing "pool size 10" in a runbook without the instance count is how the next autoscaler breaks the primary at 03:00.

Thread caps follow the pool. If each request holds one connection for the outline, there is no benefit to 200 Tomcat threads blocked on 10 connections. A thread cap near the pool size, with a short queue and a fast 503, keeps the latency of accepted work inside the budget and makes saturation visible as an error rate. The error rate is the signal to scale the bottleneck. CPU at 30 percent was the signal that sent the last team the wrong way.

Vertical move on the primary, when it is earned

If the cache miss QPS cannot be brought under the primary's measured rate, the next move is a larger primary or a read replica for the outline query, not more API nodes. A read replica helps only if the outline tolerates replication lag. A launch of a new course is the moment the row is newest and lag is most visible. The budget's freshness rule decides. "Replica lag under the spare" is a dangerous sentence, because lag is not bounded by our spare. Prefer the cache filled from the primary at publish time, which lesson 27 specifies, over a replica read that races the publish.

Practical Implementation or Design Walkthrough

Problem statement

Hold the catalog outline at p95 under 200 ms at the API during lesson 24's launch burst. The API is stateless. The outline query is serial. Enrollment badges may be omitted if they are late. We will not buy a fiftieth instance to move an 80 ms query.

Budgets and Amdahl in code

The types are checks a test can run. They do not measure production. They stop a design change from assigning 250 ms of serial hops to a 200 ms scenario, and they recompute the optimistic floor.

javaCopy
package com.techlearningpro.catalog.budget;

import java.time.Duration;
import java.util.ArrayList;
import java.util.List;

public final class LatencyBudget {

    private final Duration limit;
    private final List<Duration> serial = new ArrayList<>();
    private Duration parallelWindow = Duration.ZERO;

    public LatencyBudget(Duration limit) {
        if (limit.isNegative() || limit.isZero()) {
            throw new IllegalArgumentException("limit");
        }
        this.limit = limit;
    }

    public LatencyBudget serial(Duration hop) {
        serial.add(requirePositive(hop));
        return this;
    }

    /** Parallel hops count as one window: the max allowance, not the sum. */
    public LatencyBudget parallelWindow(Duration window) {
        parallelWindow = requirePositive(window);
        return this;
    }

    public Duration assigned() {
        Duration sum = parallelWindow;
        for (Duration hop : serial) {
            sum = sum.plus(hop);
        }
        return sum;
    }

    public Duration spare() {
        return limit.minus(assigned());
    }

    public boolean fits() {
        return !spare().isNegative();
    }

    private static Duration requirePositive(Duration hop) {
        if (hop == null || hop.isNegative() || hop.isZero()) {
            throw new IllegalArgumentException("hop");
        }
        return hop;
    }
}

javaCopy
package com.techlearningpro.catalog.budget;

public record Amdahl(double serialFraction) {

    public Amdahl {
        if (serialFraction < 0d || serialFraction > 1d) {
            throw new IllegalArgumentException("serialFraction");
        }
    }

    /** Upper bound. Ignores extra coordination as N grows. */
    public double speedup(int workers) {
        if (workers < 1) {
            throw new IllegalArgumentException("workers");
        }
        double parallel = 1d - serialFraction;
        return 1d / (serialFraction + parallel / workers);
    }
}

javaCopy
package com.techlearningpro.catalog.budget;

public final class PoolPlan {

    private PoolPlan() {}

    public static int poolPerInstance(int connectionBudget, int instances) {
        if (connectionBudget < 1 || instances < 1) {
            throw new IllegalArgumentException("budget and instances must be positive");
        }
        return Math.max(1, connectionBudget / instances);
    }
}

A test pins the launch story:

javaCopy
@Test
void outlineBudgetFitsAndFleetDoesNotBeatTheQuery() {
    LatencyBudget budget = new LatencyBudget(Duration.ofMillis(200))
            .serial(Duration.ofMillis(10))
            .serial(Duration.ofMillis(80))
            .parallelWindow(Duration.ofMillis(60))
            .serial(Duration.ofMillis(15));
    assertTrue(budget.fits());
    assertEquals(Duration.ofMillis(35), budget.spare());

    Amdahl law = new Amdahl(80d / 120d);
    assertEquals(1.5d, law.speedup(100), 0.05d);

    assertEquals(10, PoolPlan.poolPerInstance(200, 20));
    assertEquals(5, PoolPlan.poolPerInstance(200, 40));
}

assertTrue and assertEquals are JUnit Jupiter. The Amdahl assertion uses the 80 ms query over a 120 ms low-load request, matching the table. The pool assertion is the one that would have failed the autoscaler: forty instances at a budget of 200 connections get five each, not ten.

Configuration that matches the plan, for a fleet whose instance count is actually twenty:

yamlCopy
server:
  tomcat:
    threads:
      max: 50
spring:
  datasource:
    hikari:
      maximum-pool-size: 10
      connection-timeout: 80

HikariCP's connection-timeout is in milliseconds. Setting it to the outline allowance means a saturated pool fails the hop inside the budget instead of holding the request until Tomcat's longer timeout. The thread cap at 50 is a judgment: higher than the pool, enough to cover the parallel badge call that does not take a catalog connection, and far below the default 200 that would only lengthen the queue. If the badge uses a different client, it gets a different pool. One shared pool for every downstream is how the badge incident becomes an outline incident.

Decision record

ADR-104: Launch outline latency is limited by the serial query, not by API count

• Status: Accepted
• Context: Doubling API instances during a 2,700 QPS catalog burst left CPU idle and exhausted primary connections. The performance scenario is p95 under 200 ms at the API over the launch window.
• Options: (A) Keep adding API instances and raise max_connections. (B) Cache outlines so the primary sees miss traffic only, cap total pools to a benchmarked connection budget, timeout at the hop allowance, and omit a late badge. (C) Shard the catalog.
• Decision: Option B for the next launch. Connection budget 200 until a benchmark says otherwise. Pool per instance is budget / instances. No GET retry on timeout. Shard stays unaddressed; the stock and the miss rate do not require it.
• Consequences: A stale outline is possible for the TTL lesson 27 will set. Badge-less responses will be visible and must be counted. A vertical primary or a carefully lagged replica is the reopen if the miss rate cannot be held near 5 percent at burst. Raising the API count without shrinking the pool is a failed change.
• Reopen when: a benchmark at burst miss-rate shows the primary p95 above 80 ms, or product sets outline staleness to zero.

Validation

The unit test proves the arithmetic, not the percentile. The validation that matters is a rehearsal at the lesson 24 burst with the cache warmed, the pool plan applied, and timeouts counted as slow. A green p95 while the generator blocked is not a pass. Record offered QPS next to the percentile.

Expected result

The next launch review talks about miss rate, pool size per instance, and the 80 ms hop. It does not open with an instance count. Idle CPU during pain is treated as a finding, not as spare capacity.

Enterprise Real-World Scenarios

Idle CPUs, full pools

The enrollment launch at 09:00 matched the burst envelope. Autoscaling added API nodes because p95 had breached 200 ms. Each node came up with Hikari's default of 10 connections. By the time the fleet reached the mid-thirties, connection attempts failed and the new nodes contributed threads that waited. CPU charts in the incident channel were used to argue the application was healthy. Learners saw errors. The primary was not out of CPU either. It was out of slots and then out of useful work, thrashing on accept and reject.

The change that held the following launch was ADR-104's division: a hard estate-wide budget, pool size derived from the live instance count, a connection timeout inside the hop allowance, and a cache in front of the outline so the budget was spent on misses. The post-incident graph that convinced finance was not a new dashboard product. It was pool pending connections next to CPU. One of those lines explained the hour. The other did not.

A second mistake nearly survived the review. Someone proposed raising PostgreSQL max_connections to 2,000 so the defaults would fit. That spends the primary's memory on backends and moves the knee, it does not remove it. The record forbids that move without a benchmark that includes the latency percentile, not only the connection count.

Certificate PDFs and the wrong kind of scale

Certificate generation is a different path that shares the word "scale" and almost none of the structure. A PDF render is CPU-bound, takes a few hundred milliseconds, and is requested in a short ceremony at the end of a cohort, not on every page view. The first design ran it on the catalog API's Tomcat threads. A ceremony of a few hundred concurrent renders occupied the same pool the outline needs. Outlines timed out while CPU finally looked "properly" busy. The team scaled the API, which did help this particular CPU problem, and took the outline's connection budget with it.

The split that belongs in the decision: PDF rendering is a vertical or horizontal CPU problem on a separate worker fleet with its own pool of zero catalog connections. The worker pulls jobs; it does not serve the interactive budget. Throughput scales with worker CPU because the serial fraction is the render itself, and there is no shared primary on the critical path of the render. Interactive outlines keep ADR-104. Mixing them because both are "Java in the classroom app" makes every ceremony a catalog incident.

The Amdahl reading differs per path, which is the lesson. For outlines, N API nodes cannot beat the query. For PDFs, N workers can, until the object store that receives the PDF saturates, and that saturation is a different estimate. One scalability sentence for "the monolith" was the original bug.

Best Practices

• Write the scalability claim with a multiplier, a resource, and the measure that must hold.
• Find saturation before you find a bigger fleet. CPU is one signal. Queues, pool pending, and lock waits are the others.
• Treat 80 to 90 percent utilization as a latency event, not as efficient packing, until a measurement on this workload says otherwise.
• Run Amdahl on the slowest request before you buy N. If the serial fraction dominates, scale the serial element.
• Add serial budget hops. Take the max of a parallel group. Leave spare for the tail and for GC.
• Put the timeout at or inside the hop allowance, and decide the response when it fires.
• Do not retry a timed-out read into a saturated tier.
• Derive pool size from connection budget / instances. Revisit it when autoscaling changes the denominator.
• Keep interactive pools off batch CPU work. A certificate render does not get a Tomcat thread that also serves outlines.
• Count timeouts in the test percentile, and hold the offered rate at the envelope. A blocked generator measures a quieter system.
• Scale vertically when one node's CPU, memory, or disk is the proven limit and the blast radius is acceptable.
• Record the move in an ADR with the option that was only "more instances."

Common Mistakes and Anti-Patterns

Scaling the idle tier. Instances are added because latency is bad and CPU is the only graph on the bridge. The saturated pool or primary is untouched, and the new instances join the queue. The repair is to pair every latency page with the dependency's saturation signal, and to make "idle CPU" a reason to stop scaling the API.

The default pool times the fleet. Hikari's 10 and Tomcat's 200 are copied into a fleet of 40. Connections exceed the primary, or threads exceed the pool. It happens because defaults feel like someone else's capacity plan. The repair is PoolPlan in the open, tied to the instance count the autoscaler is allowed to reach.

Amdahl ignored, then misquoted. A fiftieth node is funded to shave milliseconds off a query, or horizontal scale is rejected entirely because a blog said speedup has a ceiling. The repair is the table: one request has a floor, fleet throughput still grows until the shared serial resource is busy. Name which of the two you are buying.

The budget that adds percentiles. Five downstream p99s of 50 ms are added to 250 ms and called the p99 of the page. Independent tails make the join worse than a single hop, and dependent tails can be worse still. The repair is fewer hops, a parallel window equal to the max, and a spare you do not spend in the meeting.

Retries as reliability. Clients retry timed-out GETs and multiply λ at the worst moment. The path was idempotent, so nobody called it a bug. The repair is to retry connection failure and to fail a budget overrun once.

Interactive and batch in one pool. A CPU-heavy PDF, export, or report shares Tomcat threads and JDBC connections with the outline. The ceremony becomes an outage. The repair is a worker with a queue and no interactive pool. The queue's depth is a stated bound, not an infinite buffer that hides the overload until morning.

Vertical scale as a heap trophy. RAM is doubled and the GC pause exceeds the spare in the budget. The machine is "bigger" and the percentile is worse. The repair is to measure the pause and to count it as a serial stop.

Troubleshooting and Design Review

Symptom: p95 climbs after an autoscaling event. New nodes show low CPU.

Likely cause: the new nodes added pool demand the primary cannot serve, or they added no capacity on the serial hop that was already the limit.

Investigation: pending connections, primary backend count, and the outline query time, beside API CPU. Compare instance count times pool size to the connection budget.

Corrective action: scale in, or shrink the pool per instance to the budget. Then decide whether the miss path needs a cache or a larger primary. Do not raise max_connections as the first edit.

Symptom: The mean latency is inside the budget and p99 is not, at utilization around 80 percent.

Likely cause: the knee. Variable service times or a small queue at high ρ dominate the percentile while the mean still looks acceptable.

Investigation: utilization of the bottleneck and the distribution, not only the mean. Look for retries overlapping the peak.

Corrective action: reduce ρ on that element (cache, faster query, or more of that element), and stop retrying timeouts. Buying API CPU will not move ρ on the primary.

Symptom: A rehearsal p95 looks fine and the launch does not.

Likely cause: coordinated omission in the test, a cold cache the rehearsal warmed by accident, or a launch window the test never offered.

Investigation: offered QPS versus completed QPS in the rehearsal. Confirm the cache hit ratio the test actually saw.

Corrective action: rerun at the envelope rate with timeouts scored as slow, and with a cold-miss case called out as its own scenario. A single warm run is a different workload.

Symptom: Outlines fail only while certificates are rendering.

Likely cause: a shared thread pool or a shared JDBC pool.

Investigation: which executor runs the PDF, and which pool its database calls use.

Corrective action: move renders to a worker. Protect the outline pool. Confirm a ceremony rehearsal no longer moves outline p95.

Symptom: Adding a read replica did not fix launch freshness complaints.

Likely cause: the scalability problem was lag, and the launch reads the newest course row.

Investigation: replica lag during the publish, and whether the outline cache was filled from the primary.

Corrective action: serve the new outline from a cache filled at publish, or read the primary for that key. A replica is a throughput tool for tolerant reads. It is not a freshness tool.

Design-review questions: What is the multiplier, the resource, and the measure? Which hop is serial, and what does Amdahl say at large N? Do the serial allowances plus the parallel window fit under the scenario, with spare left? What is the estate connection budget divided by the maximum instance count? What happens, in a status code, when a hop exceeds its allowance? Are timeouts retried? Which graph would show saturation if CPU is idle? Does batch work share the interactive pool?

Interview Questions and Answers

Q1. How do you define scalability so it cannot be confused with latency?

Latency is the time of one interaction on a named path, with a percentile and a window. Scalability is the ability to take a stated workload multiplier by adding a stated resource while that latency, or an error rate, stays inside its measure. A path can be fast for today's 140 QPS and unable to take the launch's 2,700 because every request needs the same primary. A path can add instances and stay slow because each request does too much serial work. I ask for the multiplier and the saturated element before I talk about shards or fleets.

Q2. When do you scale up, and when do you scale out?

I scale up when the proven limit is one node's CPU, memory, or disk, and a single writer or a single address space is still a feature. I scale out when the work splits across stateless nodes and the shared dependencies have a budget that grows correctly with N. The catalog launch was a scale-out of the API while the limit was the primary's connections and a serial query. The right moves were a cache, a pool plan, and only then a larger primary if misses still did not fit. Vertical was not "giving up." Horizontal was aimed at the wrong element.

Q3. Explain Amdahl's law with the outline numbers.

If 80 ms of a 120 ms request cannot be parallelized, the serial fraction is two thirds. Speedup is 1 / (s + (1 − s) / N). At 10 instances the upper bound is about 1.4, near 84 ms. At 100 it is about 1.5, near the 80 ms floor. Extra API nodes do not beat the query, and the bound is optimistic because it ignores the query getting slower as more nodes call it. Fleet throughput can still rise with N until that query's service rate is busy. I use the law to choose which element to buy, not as a slogan against horizontal scale.

Q4. Why is 90 percent utilization a latency problem?

In the M/M/1 teaching model, mean response is service time over (1 − ρ). At ρ of 0.9 that is ten times the service time. Real traffic is often more variable, and Kingman's approximation multiplies the wait by a variability factor as well. So "10 percent idle" is not 10 percent latency headroom. I plan the bottleneck near 70 percent of a measured capacity and I spend lesson 24's headroom there. The model is not a calculator I would quote to three digits in a capacity purchase. The shape is what I will defend.

Q5. How do you build a latency budget for a page that calls three services?

I start from the scenario's milliseconds at a named boundary. I add the serial hops. Parallel hops contribute their max, not their sum. I leave spare for tails, GC, and a late join. I set each timeout inside its allowance and I write the user-visible response when the timer fires, including a degraded body if product accepts one. I remember that several independent p99s on one request are a worse percentile than any single call. If the sum already exceeds the scenario, I cut a hop before I tune a pool.

Q6. What is the connection-pool formula you want in a review?

pool per instance = primary connection budget / instance count, using the maximum instance count the autoscaler may reach. The budget comes from a benchmark that watched latency, not from PostgreSQL's default of 100 raised until the errors stopped. Hikari's default of 10 is safe only when instance count is budget divided by 10. I set the connection timeout to the hop allowance so a full pool fails inside the budget. I keep Tomcat threads from dwarfing the pool and building a hidden queue.

Q7. Why can retries destroy a plan that was idempotent?

Idempotency means a retry does not corrupt state. It does not mean a retry is cheap. A timeout retry adds arrivals at the moment ρ is already high, where ρ / (1 − ρ) is steep. I retry connection failures, when the server likely never saw the request, and I do not retry a read that already burned the budget. Load shedding, when we need it, is a separate design. Blind retry is how a small overload becomes a large one.

Q8. CPU is at 30 percent and learners are timing out. Where do you look?

At saturation signals the CPU graph does not show: pool pending, database locks, a downstream call with no timeout, GC pauses, and thread dumps of workers blocked on a socket. I assume we scaled the wrong tier until the trace says otherwise. In the catalog incident the wait was the JDBC pool. The fix was a smaller fleet with a planned pool, plus a cache on the serial query. I would not lead with another autoscaling rule on CPU.

Q9. How do you keep a batch job from taking down an interactive path?

I give it a different process, a queue, and no connection to the interactive pool. Certificate PDFs on Tomcat threads made a ceremony into an outline outage even though "scale out" genuinely helped the PDF CPU. The interactive budget stays on ADR-104's path. The worker's scalability story is its own: CPU across N workers, until the store that receives the file saturates. One sentence of scalability for the whole JVM is how these paths get coupled.

Q10. What does a good scalability test report besides p95?

The offered rate, the completed rate, the timeout count included in the percentile, and the hit ratio of any cache. If the generator slowed down to match the system, the p95 describes a quieter workload than the launch. That is coordinated omission. I want the lesson 24 burst as the offered rate, a cold-miss case, and the pool and primary saturation metrics beside the histogram. A pretty percentile with no offered rate is not evidence.

Q11. When is a read replica the wrong scalability tool?

When the scenario's pain is freshness, or when the hot read is the row you just wrote. Replication lag is not capped by our latency spare. A replica adds read throughput for queries that can tolerate yesterday's row, or this second's lag if you have measured it. For a course outline at the moment of publish I fill a cache from the primary instead. I reach for a replica after the tolerance is written down, not because the diagram of a primary alone looked incomplete.

Q12. How do you answer "will this shard?" in a performance interview?

I go back to the envelope and the serial fraction. Catalog stock and a few thousand reads per second, mostly cacheable, do not imply a shard. The primary must survive the miss rate at a latency inside 80 ms, which is a benchmark and possibly a vertical move. Sharding answers a key-range or a write-rate problem we have not shown. I would rather defend a connection budget and an Amdahl floor than draw shards because the traffic is "launch scale."

Hands-On Exercises

Exercise: A service whose latency climbs at the peak

Problem statement. The TechLearningPro notes service returns a lesson's notes beside the outline. At Thursday's rate, p95 is 90 ms. At the launch burst, p95 exceeds 1 s, API CPU stays under 40 percent, and the notes table is on the same primary as the catalog. Write the diagnosis and the capacity move. Use the launch envelope from lesson 24 unless you label a different one.

Tasks.

1. State the performance scenario and the scalability claim, including the multiplier and the resource you are allowed to add.
2. Split a 200 ms budget. Mark one serial hop you believe is the query, and one parallel hop you could drop or degrade.
3. Compute an Amdahl upper bound for a request that is 70 percent serial. Show N = 1, 10, and 100.
4. Propose a connection budget and the pool size at 15 instances and at 45 instances.
5. Describe one retry policy and the ρ argument behind it.
6. Name the graphs that would prove saturation while CPU is idle.
7. Write an ADR that rejects "double the API fleet" as the first move, with a reopen condition tied to a benchmark.
8. Say what your design returns when the notes hop exceeds its allowance.

Expected deliverables. The scenario, the budget, the Amdahl numbers, the pool plan, the ADR, and the degraded response.

Evaluation rubric.

A recommendation that only adds instances, a budget whose serial hops already exceed the scenario, or a pool plan that ignores instance count means the diagnosis is not ready for review.

Lesson Summary

Performance is a percentile on a path. Scalability is a multiplier absorbed by a named resource while that percentile holds. The resource has to be the one that is saturated. Utilization near the top of the range multiplies latency; the M/M/1 sketch and Kingman's shape are there so 90 percent busy is treated as a queue. Amdahl's law puts a floor under a request whose slow hop is serial, and it leaves room for fleet throughput to grow until that hop runs out of service rate.

The catalog launch failed because API instances were added to a serial query and a connection budget that did not grow with them. The repair is a latency budget with timeouts inside it, a pool size of budget / instances, no timeout retries, and a cache that keeps miss traffic inside what one primary can do. Certificate renders scale on their own CPUs, on a worker that does not borrow the interactive pool. Idle CPU during a slow page is a finding.

Lesson 26 puts a load balancer in front of the horizontal copies this lesson just refused to multiply blindly, and it deals with the health checks and stickiness those copies require.

Further Reading

• Amdahl, "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities" — the bound used here as an upper limit on a request with a serial hop.
• Little's law, as in lesson 24 — mean concurrency in a stable system. The latency knee needs a queueing model beyond the identity.
• Kingman, and Jain, The Art of Computer Systems Performance Analysis — why variability and utilization compound. Use the shape in review; do not fake three digits of wait time.
• Gregg, Systems Performance — utilization versus saturation, and the habit of looking at the queue.
• Gil Tene, "How NOT to Measure Latency" — coordinated omission, and why a blocked load generator flatters the percentile.
• HikariCP documentation on pool sizing, and the PostgreSQL docs for max_connections — the defaults this lesson refuses to multiply in silence. Spring Boot's Tomcat server.tomcat.threads.max is documented with the embedded web server settings.
• Primers: /learn/high-level-design/horizontal-vertical, /learn/system-design/horizontal-vs-vertical-scaling.
• This course: /learn/advanced-system-design/capacity-estimation-and-workload-modeling, /learn/advanced-system-design/load-balancing-and-traffic-management, /learn/advanced-system-design/architecture-performance-and-capacity-review.