Architecture Performance and Capacity Review

Advanced System Design & Software Architecture · Lesson 65 · Architecture Practice, Interviews and Capstone

Lesson Overview

TechLearningPro's search page had a latency objective from Non-Functional Requirements and SLO Design: 99 percent of valid queries under 300 ms at or below 1,500 requests per second. Two weeks before enrollment, a load test on staging reported p99 of 90 ms and a green check. The test ran against an index of a thousand courses, from one load generator, with a think time that collapsed whenever the server slowed down. Production held four hundred thousand courses, a personalization call, and TLS. On the first evening of the week the page tracked the personalization service's tail, the application CPUs sat near 40 percent, and someone proposed sharding the database because "we are not scaling."

The database was not the saturated resource. The review that should have happened is the subject of this lesson. A performance and capacity review is an evidence review: a latency budget, a workload that matches Capacity Estimation and Workload Modeling, a test that does not hide the tail, a bottleneck named with utilization and saturation rather than with a hunch, and a remediation list ordered by user impact per week of work and by the bill. Sharding, a second cache tier, and a rewrite are options. They have to beat a timeout on the personalization call.

Scalability and Performance Fundamentals explained why latency bends as utilization approaches a knee. Caching Strategies and Cache Invalidation, Database Design and Indexing for Scale, and Load Balancing and Traffic Management are the usual repairs. This lesson is the review that decides which repair the evidence supports, and what it will cost to run at the next multiple of the load.

Learning Objectives

By the end of this lesson you will be able to:

1. Decompose a user-facing latency objective into a budget across hops, and state why the sum of percentile budgets is only a planning tool.
2. Use Little's law and a simple queueing model to explain headroom, with the model's assumptions labeled.
3. Distinguish open and closed load tests, and explain coordinated omission when the generator waits on the system under test.
4. Apply RED to a service and USE to a resource so a bottleneck is a saturated resource or a slow dependency, not a slogan.
5. Read a trace, a histogram, and a pool metric together before proposing a new tier.
6. Estimate the cost of a scale tactic, including a cache hit-rate change and an idle warm standby.
7. Order remediations by the SLO they buy back and the operational tax they add, and reject work that does not move the bottleneck.
8. Write the review as a decision with a reopen condition when the envelope changes.

Prerequisites

• Capacity Estimation and Workload Modeling: arrival rates, peaks, and Little's law as concurrency. This lesson uses the envelope as the test profile.
• Scalability and Performance Fundamentals: utilization, the knee, horizontal versus vertical scale.
• Non-Functional Requirements and SLO Design: the bar. A review without an SLO becomes a hunt for the largest number on a dashboard.
• Observability: Logging, Metrics and Tracing: traces and histograms are the evidence. RED and USE need the series lesson 43 can collect.
• Caching Strategies and Cache Invalidation, Database Design and Indexing for Scale, and Rate Limiting, Throttling and Load Shedding: tactics you may recommend, with their stale-data and fairness costs.
• Circuit Breaker, Retry and Bulkhead Patterns: a retry storm is a capacity event.
• Disaster Recovery and Business Continuity: the survivor region has to carry the load alone. Capacity that assumes two healthy regions is not capacity.
• Comfort with a ratio. The load tool in the walkthrough is k6's arrival-rate executor, used as a real, small script. No proprietary APM query language is required.

Detailed Core Concepts

The review asks a sequence of questions

A performance meeting that starts with "what if we shard" has already chosen a mechanism, the same failure Staff Engineer Architecture Decision-Making described. The sequence here:

1. Which SLO is missed, at which load, on which artifact? If none is missed, you are looking at a projection or a cost problem, and you say so.
2. Where does the time go on a slow request? A trace answers this. A CPU graph alone does not.
3. Which resource is utilized, saturated, or erroring? CPU, pools, disks, locks, downstream quotas.
4. Is the cause capacity, or is it work amplification: an N+1 query, a missing index, a retry, a cache stampede?
5. Which remediation moves that cause, what does it cost every month, and what quality does it worsen?
6. What do you explicitly not do?

The output is a short decision, not a twenty-page profiler dump. Profilers are evidence. They are not the review.

Latency budgets

The search journey's planning budget for a 300 ms p-threshold, written as allocations so designers can see a hog:

The allocations sum to 300 ms. That sum is a budget for design conversation. It is not a theorem about percentiles. If each hop has a p99 of its allocation, the end-to-end p99 is not the sum of the p99s. Independent tails combine less gently than a sum and more harshly than a max: a request that is slow in two hops is slower than either, and the end-to-end percentile is dominated by the heaviest tails. Use the sum to notice that personalization cannot have a 300 ms p99 inside a 300 ms journey. Prove the journey with an end-to-end histogram, the SLI from lesson 62. When a trace's critical path exceeds an allocation on real traffic, that hop is the candidate. When the sum of typical traces is fine and the tail is not, you are looking at queueing, not at a slow algorithm.

Timeouts are the budget's enforcement. Personalization at 50 ms means the client timeout is 50 ms and the page renders a default module. A timeout of two seconds "so we do not cut off a slow success" spends the entire journey on a hop the user can live without. API Gateway and Backend for Frontend is where the user-facing timeout should be the shortest honest number, with the downstream timeouts shorter than that, so the gateway returns a controlled response instead of a client abort.

Little's law, utilization, and headroom

Lesson 24 used Little's law for concurrency: the average number of requests in the system (L) equals the average arrival rate (\lambda) times the average time in system (W).

[ L = \lambda W ]

If search arrives at 200 requests per second and the average time in system is 100 ms, the average concurrency is (200 \times 0.1 = 20). That is a capacity check on pool size. A database pool of 10 connections cannot hold 20 concurrent queries; the rest wait, and (W) grows, which raises (L) further. The law does not tell you the distribution. It tells you the average occupancy you must be able to seat.

A single-server queue with exponential arrivals and service (M/M/1) makes the knee visible. If the mean service time is (S) and utilization is (\rho), the mean time in system is:

[ W = \frac{S}{1 - \rho} ]

With (S = 10) ms, service rate (\mu = 100) per second. At (\rho = 0.5), (W = 20) ms. At (\rho = 0.8), (W = 50) ms. At (\rho = 0.9), (W = 100) ms. At (\rho = 0.95), (W = 200) ms. The same server that looks "fine" at 50 percent utilization is an SLO miss at 95 percent, and nothing about the code changed. Headroom is the distance from that knee. For a latency-sensitive path, operating near 50–70 percent utilization at the documented peak is a common planning range, not a moral rule. Batch analytics can run hotter because nobody put a 300 ms SLO on it.

Label the model. Real search is not one server, arrivals are not Poisson during enrollment open, and service times are heavy-tailed. Multiple servers and caches move the knee. They do not remove it. The Universal Scalability Law, which lesson 25 introduces, adds contention and coherence costs so more nodes eventually hurt. The review uses M/M/1 as a warning about utilization, then confirms with a test at the real peak. It does not publish 63.2 percent as a magic target because a formula produced it.

Open workload, closed workload, coordinated omission

A closed workload fixes concurrency. N virtual users each send a request, wait for the response, think, and send another. When the system slows, the offered load slows with it. Throughput plateaus and the report looks stable. This matches a small room of humans. It does not match the open internet at enrollment, where learners arrive whether you are keeping up or not.

An open workload fixes an arrival rate. If the system slows, requests pile up. Latency and error graphs show the knee. k6's constant-arrival-rate executor is one implementation. A closed test that "passed" can be an open failure at the same nominal throughput, because the closed test never held the rate.

Coordinated omission, the term Gil Tene used for a measurement bug, happens when the generator or the histogram only records completions it bothered to wait for, and skips the requests it should have sent while it was blocked. The recorded latency under-represents the time real users waited. Arrival-rate executors that keep the schedule, and histograms that account for the wait to start, are how you avoid congratulating yourself. If you only have a closed test, say so in the review. Do not compare it to an SLO that was defined on real open traffic.

Also load the shape, not only the rate. Search traffic is a few popular queries and a long tail of rare ones. A test that repeats one hot query will show a fantasy cache hit rate. A test that uses a unique query per request will show a fantasy miss rate. Take a sample of production query frequencies, or state that you did not.

Data volume is part of the shape. The staging index of a thousand documents will choose different plans and different cache behavior than production. Database Design and Indexing for Scale is where a missing index hides until the cardinality arrives. A performance review that accepts a test on empty tables is not a review.

RED and USE

Tom Wilkie's RED method is for services, per request: rate, errors, duration. It matches the SLI. If duration is bad and errors are low, users are waiting, not failing. If errors climb with rate, you may be at a quota or a shed line, and lesson 62 decides whether the shed is good.

Brendan Gregg's USE method is for resources: utilization, saturation, errors. Utilization is how busy the resource is. Saturation is the queue behind it: run-queue length, pool waiters, disk queue, TCP accept queue. Errors are faults: disk errors, allocation failures, retransmits. A resource can be at 40 percent utilization and saturated if the work is waiting on a lock or a downstream and the threads are occupied. That was the search incident's shape. CPU USE was calm. The HTTP client pool's saturation — threads waiting on personalization — was the bottleneck.

The review writes one sentence: "The saturated resource is the personalization client pool; utilization of search CPU is 40 percent; the RED duration of /api/search tracks the personalization client duration." A sentence that names a product ("we need Redis") is not yet a diagnosis.

Bottlenecks in the usual order

Work amplification before new hardware. An N+1 that does 40 index lookups inside one search, a missing index that turns a lookup into a scan once the table grew, a retry that triples the load on a sick dependency, a cache stampede that expires a hot key for every course at midnight. Each of these looks like "we need capacity" and gets cheaper when you remove the wasted work. Lesson 27's stampede controls and lesson 41's retry budgets are the tactics. The review asks for the query count per request and the retry count per request before it asks for a bigger tier.

Then the saturated resource. Add capacity to the thing that is saturated, not to its neighbors. Horizontal scale of the application while the pool is waiting on one database increases the number of waiters. Scale the database, or stop calling it so often, or both, in the order the trace suggests. A read replica helps reads if the data owner allows staleness. It does not help a saturated writer. Sharding, lesson 30, is a program you start when a single primary cannot take the write rate or the data size even after indexes and amplification are fixed. It is not the first slide.

Locks and hot keys are a special saturation. A row that every enrollment updates, a cache key that every node misses together, a leader election that serializes work. More nodes make a hot key worse. The remediation is to split the key or to stop requiring that lock on the request path.

Cost of scale

Every tactic has a bill and a failure mode.

• More application instances. Near-linear cost until another tier saturates. Also more connections to the database, which can worsen pool behavior if the pool per instance stays large. Size pools so the total connections stay under the database's limit. Little's law gives you the concurrency you need. The sum of pool maxes across instances must not be a fantasy.
• Read replicas. Extra nodes plus replication lag risk, which lesson 29 priced. Useful when USE says the primary's read CPU or I/O is saturated and the SLO allows staleness.
• Cache. Memory bill, invalidation complexity, and a better hit rate's leverage. If the origin costs (C) per miss and the hit rate moves from 0.90 to 0.99, misses drop from 10 percent to 1 percent, a tenfold cut in origin load, not a ten percent cut. The arithmetic:

[ \text{misses at } 0.90 = 0.10 \lambda, \quad \text{misses at } 0.99 = 0.01 \lambda ]

A cache that moves hit rate from 0.99 to 0.995 halves the remaining origin load and may not be worth a new operational surface. Compute the absolute misses, not the vanity percentage.

• Shed and queue. Shed costs user-visible 429s, which search may spend and capture may not. A queue trades latency for survival and must have a bound, or it is an infinite W in Little's law and a memory leak with a product name.
• Regional warm standby. Lesson 64's survivor must be sized for the failover load, or the RTO is a latency incident. Paying for idle capacity is a reliability cost. Do not "optimize" it away in a performance review without reopening the RTO.
• Egress and storage. Lesson 24 showed video bytes dominating bandwidth. Search JSON is usually CPU and index I/O, not egress. Do not apply a video cost model to an API, or an API cost model to video.

The review records cost as a monthly delta next to the SLO delta. "Faster" without either number is not a recommendation.

Load profiles that answer different questions

One test cannot answer everything.

Autoscaling that reacts to CPU will not save a pool waiting on a downstream. The spike test should watch the saturated resource you named, not only CPU. If scale-out adds instances that all stampede a cold cache, the spike got worse. That result is a successful test.

Remediations are a decision

Rank by three columns: SLO time or error budget returned, engineering time, monthly money, and the quality worsened. A one-line timeout on personalization may return the search SLI in a day and worsen relevance for the slow cases. A shard may take a quarter, cost more to operate, and not move a tail that was never in the database. The staff move is to do the timeout, measure, and only then reopen. The decision record names the rejected shard and the measurement that would resurrect it: "reopen if index query p99 exceeds 150 ms at 1,500 rps after the timeout ships, with production cardinality."

Architecture and Design Diagrams

Where a search request can spend the budget:

mermaidCopy
flowchart LR
    user[Learner] --> gw[Gateway 40 ms budget]
    gw --> app[Search app 40 ms]
    app --> idx[Index 150 ms]
    app --> per[Personalization 50 ms timeout]
    per -.->|on timeout| def[Default module]
    app --> resp[Response inside 300 ms SLI]
    idx --> resp
    def --> resp

Diagnosis sequence:

mermaidCopy
flowchart TD
    slo{SLO missed at stated load?}
    slo -->|No| cost[Review cost and projection only]
    slo -->|Yes| trace[Trace the slow requests]
    trace --> use[USE on the hot resource]
    use --> amp{Work amplification?}
    amp -->|N+1, retry, stampede, bad plan| fixw[Remove wasted work]
    amp -->|No| sat[Add capacity or shed at the saturated resource]
    fixw --> measure[Re-measure end-to-end SLI]
    sat --> measure
    measure --> adr[ADR: what we will not build yet]

Queueing picture the utilization math is about:

mermaidCopy
flowchart LR
    arr[Arrivals λ] --> q[Queue grows as ρ approaches 1]
    q --> srv[Server mean service S]
    srv --> out[Completions]
    q --> lat[W = S / 1-ρ for M/M/1]

Technical Explanation

Reading a trace before a graph

A trace of one slow search, the kind lesson 43's tracing can show, lists spans with durations. Sort by exclusive time. If the personalization span is 800 ms and the index span is 40 ms, the index is not the bottleneck for that request, regardless of a dashboard someone likes. Then look at many traces, not one: a single slow GC pause is not a program. The histogram says whether the tail is a spike or a plateau. A plateau at the timeout value means almost every slow request is hitting a deadline and being cut, which is a different shape from a long slope of index queries.

Watch for missing spans. A slow call that is not traced will be blamed on whatever is traced. The review can require a span on each remote hop before it accepts a sharding proposal. That requirement is cheaper than the shard.

Pools and the hidden queue

Application frameworks put a queue in front of a pool and call the wait "latency." The USE metrics are: in-use connections divided by max (utilization), threads waiting for a connection (saturation), and timeouts (errors). A pool at 100 percent utilization with waiters is saturated even if CPU is bored. Raising the pool max without raising the database's capacity moves the queue into the database, where it becomes lock or I/O saturation and often worse latency. Compute the total: if 20 instances each hold 50 connections, you planned for 1,000 server connections. Little's law at 1,500 rps and 50 ms database time says average database concurrency is (1500 \times 0.05 = 75). A plan for 1,000 connections is several times the average and may still be right for the tail, but it is a choice you write down, not a default copied from a sample config. If the personalization pool, not the database, is waiting, raise nothing on the database.

Histograms and the SLI, again

The review uses the same bucket ratio as the SLO. histogram_quantile over a short range is how you see the shape in a meeting. Averaging p99 across pods is still wrong. Aggregate the histogram buckets, then compute the quantile or the threshold ratio. A pod serving 1 percent of traffic must not count equal to a pod serving 40 percent.

k6 as an open workload

This script holds an arrival rate. It is a real k6 API. It does not, by itself, create production-like query diversity; the function under test has to pick queries from a file of real shapes.

javascriptCopy
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  discardResponseBodies: true,
  scenarios: {
    search_peak: {
      executor: 'constant-arrival-rate',
      rate: 1500,
      timeUnit: '1s',
      duration: '15m',
      preAllocatedVUs: 200,
      maxVUs: 2000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<300'],
  },
};

export default function () {
  const res = http.get('https://staging.example/api/search?q=concurrency');
  check(res, { 'status 2xx': (r) => r.status >= 200 && r.status < 300 });
}

preAllocatedVUs and maxVUs have to be high enough to sustain the rate when latency rises, or k6 will drop iterations and you are back to a softened workload. The threshold p(99)<300 is a gate for this run, not a 30-day SLO. Say that in the report. discardResponseBodies is a generator fairness choice so the client does less work; do not use it if you need to assert on the body. The URL's constant query string is the fantasy hot key. A serious run reads a distribution. Staging that cannot take 1,500 rps is not a reason to test at 50 and multiply; it is a reason to test at the highest safe open rate and label the extrapolation as an assumption.

Cost sketch for the search decision

Assume, labeled, the index tier costs 4,000 currency units a month and is at 40 percent CPU at peak. Personalization is a separate service with a tail. A new cache tier would cost 1,500 a month plus an on-call integration. It would reduce index load only if the tail were index misses. The trace says it is not. The timeout costs no new tier. The relevance regression is the price: searches where personalization exceeds 50 ms lose a module. Product accepts that, in writing, because the alternative is the whole page missing 300 ms.

If after the timeout the index p99 at 1,500 rps is 120 ms, you do not buy the cache yet. If a projection to 4,000 rps, three years out, would push index CPU through 70 percent at peak, you put that projection in the ADR as the reopen, and you do not buy the hardware in this quarter. Capacity you buy years early is a cost with no SLO delta.

Headroom after failover

Region B in lesson 64 is sized as a warm standby. If it runs at a quarter of the instance count, a failover during enrollment fails the latency SLO even though the database promoted cleanly. The capacity review and the DR review share a sentence: the survivor's open-rate test at peak is part of the RTO, or the RTO is only a data-plane number. Idle capacity to pass that test is the cost of the regional objective. Cutting it to save money reopens lesson 64.

What "good" evidence looks like

A packet a staff engineer can accept has: the SLO and the envelope, the test type and its limitations, the histogram at the gateway, one trace of a typical slow request, USE for the accused resource, the query or call count per request, the monthly cost of the top two options, and the option refused. A packet of screenshots without units, or a single average, goes back. So does a packet whose test data is a thousand rows when production has orders of magnitude more, unless the claim is explicitly limited to code-path regressions.

Practical Implementation or Design Walkthrough

Problem statement

Enrollment week is close. Staging's closed test says search is fine. Production RED shows p99 near 700 ms at 800 rps, under the shed line of 1,500, with CPU at 40 percent. Recommend a remediation order. Do not start a shard.

Evidence you insist on

You pull, or you refuse to decide without:

• Gateway histogram for /api/search over the last peak hour, aggregated buckets, fraction under 300 ms.
• A sample of traces above 300 ms. In this walkthrough they show personalization at 400–900 ms, index at 30–80 ms, application at 15 ms.
• Personalization client pool: 100 percent of the 20 connections in use, 30 threads waiting, timeout count climbing.
• Index CPU 40 percent, no disk queue, buffer cache hit rate high. These are USE numbers for the resource someone wants to shard.
• Call count: one index query per search, not N+1. Retry count on personalization: three attempts, which multiplies the occupancy. That retry is work amplification.
• Staging test description: closed, 50 VUs, 1,000 documents, one query string. You discard it as evidence for the SLO and keep it only as a regression smoke test.

Options

A. Status quo. Misses the SLO at half the documented peak. No cost change. Rejected.

B. Disable personalization retries and set a 50 ms timeout; render the default module. One change in the client. Journey latency returns to the index plus the application, which the traces say is inside 150 ms for the body. Relevance loses the module when personalization is slow. No new tier.

C. Scale personalization and keep the call synchronous with a 300 ms budget. Moves the bottleneck only if personalization can actually get faster at this rate. Unknown cost until their USE is reviewed. Does not belong as the first bet if a timeout already fits the product decision, because their saturation may be a downstream you do not own.

D. Add a response cache in front of search. Would hide repeated queries. The slow traces are slow because of personalization, which is per learner and caches poorly. Hit-rate math does not pay. Rejected for this incident, reopen if index time becomes the span.

E. Shard the index. Write rate and data size do not demand it. The saturated resource is not the index. Rejected. Reopen condition stated below.

The decision

Accept B. Remove the retry so a slow personalization call occupies one slot, not three. Timeout at 50 ms, which is the budget line. Measure the gateway SLI at the next peak. Keep the shed line at 1,500 rps. Open a follow-up for personalization's own USE review; it is their service, and search will no longer wait for it.

Reopen D or a capacity increase on the index if, after B, index span p99 exceeds 150 ms at 1,500 rps on production cardinality. Reopen E only if a single index primary's write or size limit is the evidence, which is not this packet.

Validation

The next open-rate test in staging uses a restored production-sized index or a documented fraction, a query file drawn from real frequencies, constant-arrival-rate at the highest staging can take, and a personalization stub that sleeps 800 ms for 20 percent of calls. The assertion is that the gateway duration stays within the SLO and the body still returns. A test that stubs personalization at 1 ms does not prove B. After release, the lesson 62 ratio over the enrollment peak is the real result. One staging run is a rehearsal.

Expected result

Search meets the 300 ms fraction at current peak without a new data tier. The shard proposal is an ADR rejection with a reopen condition, so it does not return as a rumor. Personalization's team gets a trace, not a blame paragraph. Finance does not pay 1,500 a month for a cache that the span data says would not be on the critical path.

Enterprise Real-World Scenarios

The green load test and the empty index

The staging test had been a release gate for a year. It used fifty virtual users and a fixture catalog. Every release passed. Production cardinality crossed the point where the planner stopped using the index that the fixture always used, because the statistics on a thousand rows and on four hundred thousand rows are not the same. The query was still "correct." It scanned. CPU on the index rose only into the sixties, and latency doubled, and the gate stayed green because nobody had asked the gate to use an open rate or a realistic volume.

The review's first action was to demote that test from "SLO evidence" to "smoke." The second was the personalization timeout, which was the actual production tail once traces were read; the scan was real and secondary, and it got an index fix the same week because the call count and the plan were cheap to prove. The third was a quarterly restore of a masked production index into the performance environment. Masking mattered because lesson 63's personal data must not be copied into a test account casually. The cost of the larger staging index was accepted as the cost of believable evidence. A shard design that had been staffed from the green test was stopped before it started. The reopen condition was written against production span times, not against the fixture.

The social part of the review was to say, without theater, that the gate had been measuring the wrong system. People had acted in good faith on a green check. The repair was the definition of evidence, not a hunt for who wrote the fixture.

Scale-out that fed a retry storm

A different peak, on the enroll path, showed CPU at 70 percent and rising latency. Autoscaling added instances. Latency rose again. Traces showed the catalog client retrying three times with no budget, and each new instance joining the retry. Lesson 41's pattern was in the code "for resilience" and had no upper bound. USE on catalog showed saturation of its CPU; RED on enroll showed duration and then errors. Adding enroll instances increased catalog's arrival rate by the retry factor, about three, which pushed catalog further into the knee, which created more timeouts, which created more retries.

The capacity review refused more enroll pods. The remediation was a retry budget and a bulkhead so catalog slowness could not occupy every enroll thread, plus the synchronous-versus-outbox distinction from lesson 61 for anything enroll did not need in the response. After the retry budget, catalog CPU fell and enroll latency followed, with no additional instances. The autoscaler rule that looked at CPU alone was left in place for genuine traffic growth and was no longer the incident response for a saturated downstream. The cost avoided was a doubled enroll tier that would have made the next storm larger. The principle is the one the M/M/1 sketch is for: past the knee, more offered load increases W sharply, and retries are offered load you created.

Best Practices

• Start from the missed SLO and the load. If the SLO is met, say you are reviewing cost or a projection, not an incident.
• Allocate a latency budget so one hop cannot legally consume the journey, then enforce it with timeouts shorter than the user wait.
• Treat the sum of hop percentiles as a sketch. Judge the journey on an end-to-end histogram.
• Ask for an open arrival-rate test at the documented peak, with production-like cardinality and query mix. Label anything less.
• Diagnose with RED on the service and USE on the resource. Name the saturated resource in one sentence before naming a product.
• Remove amplification — extra queries, unbounded retries, stampedes — before you add tiers.
• Size pools from Little's law and a written total-connection ceiling. Do not copy pool defaults per instance.
• Compute cache value from absolute misses removed, not from a hit-rate vanity point.
• Put monthly cost next to SLO delta. Idle failover capacity is a DR cost; do not cut it in this review by accident.
• Reject mechanisms that do not touch the bottleneck, in writing, with a reopen measurement.
• Include the survivor region in the capacity envelope if the RTO assumes failover during peak.
• Prefer a small packet: histogram, trace, USE, options, decision. A profiler novel is an appendix.

Common Mistakes and Anti-Patterns

The average as the result. The report leads with mean latency. The SLO is a tail fraction. Means hide the queue. The repair is the gateway histogram ratio and a tail trace.

Closed test, open traffic. Virtual users back off when the system slows, the chart plateaus, and production does not. The repair is an arrival-rate test, and a sentence in the report when you could not run one.

Coordinated omission. The generator records only the requests it completed after waiting, so the outage looks faster than users experienced. The repair is a generator that keeps the schedule and a histogram that includes delayed starts. Name the limitation if the tool cannot.

Scaling the wrong tier. Application pods multiply while they wait on one pool. It happens because CPU autoscaling is the tool at hand. The repair is USE on the pool and a stop to scale-out until the downstream is bounded.

Shard as a reflex. Partitioning is proposed for a read-tail that is a remote call. It happens because sharding is the advanced-sounding option. The repair is the diagnosis sequence and an ADR that says what would make sharding real.

Fantasy hit rates. A cache is justified with "we will get 99 percent" on a per-user personalized page. Absolute misses do not fall. The repair is the miss arithmetic and a trace that shows whether the slow span is cacheable.

Fixture data. The test passes on a thousand rows. The planner changes at production cardinality. The repair is a masked production-sized run, or an explicit non-claim.

Headroom as waste. Someone removes the idle region-B capacity because utilization is low. The next failover misses the latency SLO at peak. The repair is to review that capacity against the DR document, not against a weekly CPU average.

Troubleshooting and Design Review

Symptom: p99 is an order of magnitude above the median, CPU is moderate.

Likely cause: queueing at a lock, a pool, or a slow dependency with a heavy tail. Not "Java is slow."

Investigation: traces of requests above the SLO, pool wait metrics, downstream RED. Check retry multipliers.

Corrective action: timeout or bulkhead on the slow hop, then remove unbounded retries. Re-measure before adding nodes.

Symptom: Adding instances does not move the SLI.

Likely cause: the saturated resource is shared: one database, one hot key, one external quota.

Investigation: USE on that shared resource during the scale-out. Connection counts should rise if the app is the waiter.

Corrective action: stop scaling the waiter. Fix the shared resource or stop calling it on the critical path.

Symptom: Staging passes and production fails at half the rate.

Likely cause: data volume, query mix, TLS, personalization, or a closed generator.

Investigation: compare cardinality, plans, and test type to production traces.

Corrective action: change the evidence standard. Do not tune production from the fixture result.

Symptom: The cache hit rate is 98 percent and origin CPU is still on the knee at peak.

Likely cause: 2 percent of a large (\lambda) is still a large miss rate, or misses are the expensive queries while hits are cheap health checks.

Investigation: absolute misses per second, and whether the hit counter includes a trivial route.

Corrective action: price absolute origin load. Exclude health checks from the hit-rate story. Scale or reshape the misses that remain.

Symptom: After failover, latency SLO fails though lag is zero.

Likely cause: region B was not sized or warmed for peak. Caches are cold. Connection pools thundering-herd the new primary.

Investigation: compare instance counts and cache hit rates to the capacity envelope. Look at the first minutes after promote.

Corrective action: size and test the survivor at peak. Stagger pool creation. This is a joint finding with the DR plan, not a reason to abandon fencing.

Design-review questions: Which SLO, at what load, is the subject? Open or closed test, and what data volume? Where does a slow trace spend exclusive time? What is utilized, saturated, and erroring? What is the call count and retry count per request? What does the remediation cost per month, and which quality gets worse? What did you refuse, and which measurement brings it back? Can the failover region carry this envelope?

Interview Questions and Answers

Q1. How do you build a latency budget, and what do you not claim about it?

I start from the user-facing threshold, for example 300 ms for search, and I allocate it across gateway, application, the necessary data call, and optional dependencies, with a timeout on anything optional. The allocations are a design conversation so one hop cannot consume the journey. I do not claim that summing per-hop p99 values yields the end-to-end p99. Tails do not add that cleanly. The number I will defend is the end-to-end histogram at the gateway, under the stated load. The budget's job is to force a 50 ms timeout on personalization instead of a two-second wait.

Q2. Why does latency explode as utilization climbs, even if the code is unchanged?

In a simple M/M/1 model, mean time in system is (S / (1 - \rho)). At 50 percent utilization a 10 ms service takes 20 ms in system; at 90 percent it takes 100 ms; at 95 percent it takes 200 ms. Queues, not instructions, ate the budget. Real systems have more servers and heavier tails, so I label the model as a warning and confirm with an open test at peak. The practical consequence is headroom on latency-sensitive paths and a refusal to run the search tier at 95 percent because average CPU "looks acceptable."

Q3. What is coordinated omission?

It is a measurement failure where the load generator or the histogram only records work it successfully waited for, and omits the requests it should have issued while it was blocked. The chart then under-reports how long real users waited. It shows up in closed-loop tests that slow their sending when the server slows. I prefer an arrival-rate executor that keeps the schedule, and I state the limitation when the tool cannot. A green p99 from a generator that backed off is not evidence the SLO will hold at a fixed 1,500 rps.

Q4. RED versus USE: when do you use each?

RED is rate, errors, and duration for a service. It tells me whether the user-facing SLI is actually missed and whether we are failing or merely slow. USE is utilization, saturation, and errors for a resource: CPU, a pool, a disk, a lock. I use RED to pick the service and USE to name the bottleneck. The search incident was bad RED duration with calm CPU utilization and a saturated client pool. A review that only watches CPU will scale the wrong tier.

Q5. The database CPU is at 40 percent and someone wants to shard. What do you ask?

I ask which SLO is missed and which span is slow. If traces show a downstream call at hundreds of milliseconds and one cheap index query, sharding does not touch the bottleneck. I ask for N+1 counts, plans at production cardinality, and lock or pool saturation. Sharding is justified when a single primary cannot sustain the write rate or the data size after amplification is removed. I record the refusal and a reopen condition in numbers, so the idea can return when the evidence changes and not before.

Q6. How do you use Little's law in a capacity review?

Average concurrency is arrival rate times time in system. At 1,500 searches per second and 50 ms on the database, average database concurrency is 75. That sizes pools and tells me whether a 10-connection pool will queue. It does not give me the p99. I also check the total of per-instance pool maximums against the database's connection limit, because twenty generous pools are a self-inflicted saturation. If time in system grows because of a downstream wait, Little's law tells you occupancy will grow with it; it does not tell you to add application CPU.

Q7. A cache vendor promises to take you from 90 percent to 99 percent hit rate. How do you evaluate it?

I compute absolute misses. At hit rate 0.90, misses are (0.10\lambda); at 0.99 they are (0.01\lambda), a tenfold origin reduction, which can be worth a tier if origin is the saturated resource and the responses are cacheable. Moving from 0.99 to 0.995 only halves the remaining misses and may not pay for a new system. Personalized search often will not hit at all. I want the trace to show the slow span is a cacheable call, and I want the invalidation story from lesson 27, including the staleness the SLO allows. A hit rate that includes health checks is not a number I will use.

Q8. What is in a performance packet you are willing to accept?

The SLO and the load envelope, whether the test was open or closed, the data volume and query mix, the aggregated histogram, a representative slow trace, USE for the accused resource, retries and query counts, the monthly cost of the leading options, and the option refused with a reopen measurement. Screenshots of averages, or a test on a thousand-row fixture offered as proof of the production SLO, go back. I would rather decide on a smaller packet that names the bottleneck than on a large packet that does not.

Q9. How do retries and autoscaling interact at the knee?

Past the knee, added load increases latency sharply. A client that retries timeouts multiplies load on a sick dependency, which moves it further past the knee. An autoscaler that adds callers because their CPU rose, or even because they are busy waiting, multiplies the retries again. I cap retries, bulkhead the dependency, and scale the saturated resource rather than the waiters. The enroll incident in this lesson got worse every time the autoscaler helped. The fix was a retry budget, after which extra pods were unnecessary.

Q10. How do you connect this review to disaster recovery?

The region that receives a failover has to meet the latency SLO at the enrollment envelope, not at idle. If warm standby runs a fraction of the instances, the RTO's "we failed over" can still be a performance incident. I include a peak open-rate test against the survivor, or I label the RTO as data-only. Idle capacity that exists for that test is not waste to delete in a cost pass. Deleting it reopens the DR decision.

Q11. You have one week before a traffic spike. What do you do first?

I read traces and the pool metrics for the journey that has an SLO, and I look for a timeout, an unbounded retry, or an N+1 I can remove without a migration. I confirm the shed line is real for routes that are allowed to shed, and I confirm capture is not behind that shed. I run the best open test the environment allows and label its gaps. I do not start a shard or a new cache tier in that week unless the trace says the slow span is the origin and a cache is already built. The goal is to move the known bottleneck and to know what I could not prove.

Hands-On Exercises

Exercise: Review performance evidence and prioritize remediation

Problem statement. You are given a search path that misses a 300 ms objective at 800 rps. CPU is 40 percent. A teammate proposes sharding the index and adding a cache. Staging's closed test is green. Write the review: what evidence you still need, what you think is likely, which remediations you order, and which you refuse. You may invent measurements only if you label them as assumptions.

Tasks.

1. State the SLO, the envelope, and whether 800 rps is inside it.
2. List the minimum evidence: histogram, trace, USE, test method, data volume.
3. Apply Little's law to one hop using an assumed service time you label.
4. Sketch an M/M/1 or qualitative knee argument for why 40 percent CPU does not acquit the design.
5. Propose a latency budget with one timeout.
6. Rank at least three remediations, including the shard, with cost shape and the quality each worsens.
7. Write the decision and the reopen condition.
8. Say how the test must change so the next green check is evidence.

Expected deliverables. A review memo of two to four pages, a budget table, and a short ADR rejection or acceptance.

Evaluation rubric.

A memo that says "shard and cache for scale" with no trace and no envelope is not a review.

Lesson Summary

A performance and capacity review is a decision about evidence. The SLO and the workload envelope define the bar. Traces show where slow requests spend time. RED says whether users feel errors or delay. USE names the saturated resource, which may be a pool at 40 percent CPU. Little's law sizes concurrency. A single-server queue shows why headroom exists, under assumptions you must say out loud. Open tests at realistic cardinality are the experiments; closed tests on fixture data are smoke.

Remediation starts with wasted work and timeouts on optional hops, then capacity on the resource that is actually saturated, then new tiers when the miss arithmetic pays. Sharding and caches that do not touch the slow span are written down as refused, with a measurement that can bring them back. Cost sits next to the SLO delta, including the idle capacity a failover region needs if the recovery plan is real.

This module's practice is now a set of reviews you can run on one system: the decision frame, the service level, the threat model, the recovery objectives, and the capacity evidence. The capstone that follows in this course is where those packets have to agree.

Further Reading

• Brendan Gregg, Systems Performance — the USE method and the habit of naming a saturated resource before buying a tier.
• Gil Tene, "How NOT to Measure Latency" — coordinated omission and why closed-loop averages lie about user wait.
• Little's law — occupancy as (\lambda W), used here as a pool check, not as a percentile predictor.
• The M/M/1 mean sojourn (S/(1-\rho)) — a labeled illustration of the knee. Lesson 25's Universal Scalability Law is the multi-node extension.
• Tom Wilkie's RED method — rate, errors, duration, aligned with the SLI in lesson 62.
• k6 arrival-rate executors — one concrete open-workload tool. The script in this lesson is the public executor API.
• Gunther, Guerrilla Capacity Planning — practical skepticism about benchmarks that do not match production mix.
• This course: Capacity Estimation and Workload Modeling, Scalability and Performance Fundamentals, Non-Functional Requirements and SLO Design, Observability: Logging, Metrics and Tracing, Caching Strategies and Cache Invalidation, Database Design and Indexing for Scale, Circuit Breaker, Retry and Bulkhead Patterns, Disaster Recovery and Business Continuity.