Non-Functional Requirements and SLO Design

Advanced System Design & Software Architecture · Lesson 62 · Architecture Practice, Interviews and Capstone

Lesson Overview

TechLearningPro's quarterly reliability slide said both course search and payment capture were "highly available, targeting three nines." Search is a read of an index that may be a minute stale. Capture moves money and must not be retried into a second charge. The same sentence had been copied into both runbooks. During enrollment week, search shed load with 429 and the page stayed useful. The same shed, applied by a shared library to capture, turned a traffic spike into abandoned payments and a reconciliation backlog. The adjective was identical. The service level was not.

"Non-functional requirement" is the name enterprises use for the qualities that are not a user story. The name is unfortunate, because it sounds optional. Architectural Thinking and Quality Attributes already replaced adjectives with six-part scenarios: source, stimulus, artifact, environment, response, response measure. This lesson is what you do with the response measure after the design is running. You turn it into a service level indicator, a service level objective, and an error budget that a release process is allowed to spend. You keep a service level agreement in the commercial bucket it belongs to, so an internal target is not confused with a credit clause.

The arithmetic is part of the design. A 99.9 percent availability objective over 30 days is 43.2 minutes of budget, not a mood. A dependency chain of three independent 99.9 percent services is about 99.7 percent at the user, and only under an independence assumption you must label. A latency objective without the workload from Capacity Estimation and Workload Modeling is a laptop result. Module 3's load shedding, replication, and idempotency lessons are the mechanisms you spend budget on. This lesson decides the budget and where it is measured.

You will write SLOs for TechLearningPro payment capture and product search, show the queries that implement the indicators, and leave a policy for what the team does when the budget is gone.

Learning Objectives

By the end of this lesson you will be able to:

1. Rewrite an adjective NFR as a six-part scenario whose response measure can be collected on a named artifact.
2. Define an SLI, an SLO, and an SLA so a contract credit cannot be mistaken for an engineering target.
3. Compute an availability error budget in minutes and in failed requests, and state which definition you are using.
4. Specify a latency SLI as a fraction of requests inside a threshold, and explain why averaged percentiles are not that fraction.
5. Estimate a user-journey objective from component objectives only as a labeled independence assumption, and prefer a journey SLI measured at the edge.
6. Attach every SLO to a workload envelope so the environment slot of the scenario stays honest.
7. Write different objectives for payment capture and product search, including what each path is allowed to shed.
8. Describe a burn-rate alert and an error-budget policy that changes release behavior when the budget is consumed.

Prerequisites

• Architectural Thinking and Quality Attributes: the six slots. This lesson operationalizes the response measure. It does not replace the scenario.
• Requirements and Architecture Drivers: an SLO that no stakeholder will trade against a feature is a wish.
• High-Level System Design Methodology and Capacity Estimation and Workload Modeling: the environment and the envelope. Estimates stay labeled assumptions until a counter replaces them.
• Scalability and Performance Fundamentals: why a system near saturation misses a latency SLO even though average CPU looks fine. The review practice is lesson 65.
• Rate Limiting, Throttling and Load Shedding, Database Replication and Consistency, and Idempotency and Exactly-Once Processing: tactics that either protect a budget or silently burn it.
• Observability: Logging, Metrics and Tracing: timers and traces are how an SLI exists. This lesson specifies which series count.
• Comfort with a ratio and with a Prometheus range vector. The queries use Micrometer's default HTTP timer names. No custom metrics vendor is introduced.

Detailed Core Concepts

From adjective to scenario to objective

An NFR spreadsheet that says "availability: high" cannot fail a design review, because nothing counts as a miss. Lesson 2's scenario can. A search scenario this course will accept:

• Source: a signed-in learner.
• Stimulus: submits a course query.
• Artifact: the search API at the gateway.
• Environment: a weekday evening at the modeled peak from lesson 24, index lag under sixty seconds, one availability zone impaired.
• Response: the API returns a result page or an explicit shed.
• Response measure: over 30 days, at least 99.9 percent of valid queries complete in under 300 ms, and at least 99.9 percent return a non-5xx status.

That measure is still a sentence. An SLI is the time series that implements it. An SLO is the target you set on that series, plus the window. The scenario remains the architectural claim. The SLO is the operating contract with the same numbers, written so an on-call dashboard and a release policy can read it. If the scenario and the SLO diverge, fix one of them in the open. Do not let the dashboard become a quieter document than the design review.

"Valid query" is part of the SLI, not a footnote. Requests that fail authentication are the identity provider's behavior and the client's bugs. Count them in a different indicator. Requests that your gateway rejects as oversized are your policy working. Decide, in writing, whether they consume the search budget. The decision matters more than the percentage you pick.

SLI, SLO, SLA, error budget

These four terms are the whole vocabulary. Keep them in separate sentences so a sales deck cannot merge them.

An SLI is a quantitative measurement of some aspect of the service, defined as a ratio of good events to valid events whenever you can. Availability as "the fraction of valid HTTP requests that did not return 5xx and did not time out at the gateway" is an SLI. "The site is up" is not.

An SLO is a target on an SLI over a window. "99.9 percent of valid search requests in each rolling 30 days are good, and 99 percent of them finish in under 300 ms" is two SLOs. The window is part of the objective. A target with no window can be satisfied by a lucky hour.

An SLA is a commercial agreement: if the measured service falls short of a stated level, a named remedy applies, often a credit or a right to terminate. The SLA's number should be looser than the SLO. You burn the gap between them on ordinary mistakes without writing a credit. If your SLO is 99.9 percent and your contract says 99.9 percent, you have no room to be an engineer. Many internal platforms have SLOs and no SLAs at all. That is a fine outcome. Inventing an SLA to sound serious creates a financial obligation nobody priced.

An error budget is what the SLO allows you to get wrong. For an availability SLO of 99.9 percent, the budget is 0.1 percent of the window. Over 30 days of 24 hours:

[ 30 \times 24 \times 60 = 43{,}200 \text{ minutes} ]

[ 0.001 \times 43{,}200 = 43.2 \text{ minutes} ]

The same objective at 99.95 percent is 21.6 minutes. At 99.99 percent it is 4.32 minutes. At 99.999 percent it is 25.9 seconds. Those figures are why "add another nine" is an architecture program, not a configuration tweak. Detection, failover, and human response from Disaster Recovery and Business Continuity have to fit inside the budget or the objective is fiction.

Request-based budgets use counts. If search serves (10^7) valid requests in the window and the SLO is 99.9 percent good, the budget is 10,000 bad requests. A one-minute total outage at peak can consume more of this budget than a one-minute outage at dawn. Time-based availability treats those minutes as equal. Neither definition is wrong. They answer different questions. Payment care about failed capture attempts, so a request-based SLI matches the user harm. A status page that promises "the product is reachable every minute" matches a time-based SLI. Write which one the dashboard implements. A team that computes a request ratio and calls it uptime will argue with itself during the incident.

The Google SRE book (Beyer, Jones, Petoff, Murphy) is the reference that fixed these meanings in the industry. The SRE Workbook adds the alerting policy. This lesson uses those meanings and does not invent a private dialect.

Latency SLIs and the percentile trap

"Average latency under 100 ms" hides the tail the learner actually hits. Lesson 2 already requires a percentile or an equivalent concrete check. The equivalent this lesson prefers, because it has an error budget, is:

The fraction of valid requests whose total time at the gateway is ≤ 300 ms is at least 0.99 over the window.

For a single set of requests, that sentence is the same claim as "p99 ≤ 300 ms." The operational difference shows up when people aggregate. The average of sixty per-minute p99 values is not the p99 of the hour. A quiet minute with three slow calls can show a terrible p99 and almost no user harm. A busy minute with a slightly elevated p99 can burn the real budget. Alerting on the slow-request fraction, with traffic as the weight, tracks the SLO. A chart of p99 remains useful in a review. It is a bad paging signal by itself, and it is a bad number to average across instances or minutes.

Histograms make the fraction computable. Micrometer's HTTP timer, scraped by Prometheus, exposes a cumulative histogram: a _bucket series with a le label. The fraction of search requests at or under 200 ms over five minutes is a ratio of rates:

promqlCopy
sum(rate(http_server_requests_seconds_bucket{uri="/api/search",le="0.2"}[5m]))
/
sum(rate(http_server_requests_seconds_count{uri="/api/search"}[5m]))

The buckets have to exist at the threshold you care about. A histogram whose largest finite bucket is 100 ms cannot answer a 300 ms objective; everything slower looks identical. Set the buckets from the SLO, not from a framework default you have not read. rate over a short range is an alert view. The SLO itself needs a recording rule retained for the whole window. A Prometheus server that keeps fifteen days of raw samples cannot answer a 30-day objective unless you recorded the good and valid counts as a slower series and stored that series long enough. The SLI has a storage requirement. That requirement is an architecture decision, small and easy to skip.

Client timeouts create a second clock. If the browser gives up at 2 seconds and your server records an 8-second success, you counted a good event the learner experienced as a failure. Measure at the gateway the user is waiting on, and decide whether a client-cancelled request is bad. API Gateway and Backend for Frontend is where that clock usually sits.

Journey SLIs versus component SLOs

Component objectives are how teams work. Users experience a journey: search, open a course, enroll, pay. If three components must all succeed and their failures are independent, the journey success ratio is the product of the component ratios. Three services at 0.999:

[ 0.999^3 = 0.997003 ]

About 99.70 percent, which is 129.6 minutes of budget in a 30-day time-based reading, not 43.2. The product is a design-time estimate. It is valid only while independence holds. Shared databases, a shared zone, and a shared deploy make failures correlated. Perfect correlation, where the three fail together or not at all, pulls journey availability back toward the single-service number for that shared fault, while leaving independent faults to multiply. You cannot know the mix in advance. The staff rule is: publish the product as a labeled assumption when you are sizing, and measure the journey at the edge for the SLO you will actually defend.

There is a second trap. Multiplying objectives is not the same as multiplying achieved SLIs. If every team sets 99.9 and lives at 99.99, the journey is finer than the product of the targets. If every team treats 99.9 as a ceiling they bump into, the product is optimistic. Read achieved SLIs when you argue about a user complaint. Read objectives when you argue about a design that does not exist yet.

Payment adds a further cut. A capture call that returns 200 after the acquirer timed out and the outcome is unknown is not a good event, even though the HTTP status was successful. The SLI for capture counts terminal, reconciled outcomes, not TCP success. That definition depends on idempotency keys and an explicit unknown state, which lesson 37 and Design a Payment Processing System already require. An availability SLI that counts 200 will show a healthy payment service on the worst day of the quarter.

Durability is not availability

Availability asks whether the service answered just now. Durability asks whether an acknowledged write is still there after a fault. A database can answer every health check and have lost the last second of commits, if the primary acknowledged a transaction before the WAL reached a standby. Lesson 29's synchronous_commit values are the mechanism. The SLO has to name the fault. "No acknowledged enrollment is missing after the loss of one availability zone" is a durability SLO. "We have eleven nines because the cloud vendor said so" imports an object-storage annual-loss figure into a database conversation where it does not apply.

Durability budgets are uncomfortable because the acceptable number of lost acknowledged writes is often zero inside a stated fault model, and nonzero only for a wider disaster. That split belongs in the DR lesson: the SLO says what "good" means, and the RPO says how much a disaster is allowed to rewind. Do not use one word for both.

The workload is part of the objective

A latency SLO without a load is incomplete, because the environment slot was doing real work in lesson 2. Search at 50 requests per second and search at 2,000 requests per second are different designs. Scalability and Performance Fundamentals shows why latency bends upward as utilization approaches the knee. Write the envelope next to the SLO:

• Steady evening: 200 searches per second, p99 under 300 ms.
• Enrollment-week peak: 1,500 searches per second for up to two hours, p99 under 500 ms, explicit 429 allowed beyond that arrival rate.
• Above the shed line, the availability SLI counts a 429 with a Retry-After as shed, not as a server error, for search only.

The shed line is a product decision. It is also how you keep a latency SLO from forcing infinite capacity. Lesson 24's assumption table supplies the rates. When a measured peak exceeds the table, you either raise capacity, lower the SLO, or move the shed line, and you record which one in the same way Staff Engineer Architecture Decision-Making records any other trade.

Error-budget policy

An SLO that does not change anyone's week is a report. The policy is the point. A usable policy for this course:

• While more than half the 30-day budget remains, feature work proceeds. Reliability work is still scheduled; it is not an emergency.
• When a fast burn is underway, the on-call response is to stop the bleed: roll back, shed load where the SLO allows it, or disable a non-critical dependency. New risk waits.
• When the monthly budget is exhausted, releases that add risk pause. The team spends the time on the items that burned the budget: a bad query, a retry storm, a dependency with no timeout. The pause has an expiry and an owner, so it cannot become a permanent freeze nobody is working.
• SLO misses caused by a vendor you do not control still count. The policy response may be a bulkhead or a contract change. Excluding them by habit trains the dashboard to go green on the user's worst path.

Burn-rate alerts exist so you do not wait until day 30 to notice, and so you do not page on a single blip. The SRE Workbook's approach pairs a long window with a short confirmation window. A burn rate of 14.4 consumes about 2 percent of a 30-day budget in one hour. For a 99.9 percent objective that is an error rate of about 1.44 percent, because (14.4 \times 0.1% = 1.44%). A burn rate of 6 over six hours consumes about 5 percent of the budget. Page when both the long window and a short window exceed the threshold, so an already-finished spike does not wake someone. Ticket, rather than page, when a slower burn will exhaust the budget in a few days. The exact windows belong in the runbook next to the query. The principle belongs here: alert on budget consumption rate, not on a host being down.

Budget accounting has a reset. A rolling 30-day window forgets old incidents gradually. A calendar-month window resets on the first, which can encourage risky releases on the second and panic on the twenty-eighth. Rolling windows match user memory better. Calendar windows match business reviews better. Pick one and do not graph the other in the same meeting without labeling it.

What you refuse to put in an SLO

User satisfaction, "no SEV-1s," and "architecture quality" are not SLIs. They are outcomes influenced by many things, including how loudly a single customer escalates. A SEV-1 policy can sit beside an SLO. It is not a substitute, because a stream of ten-minute pains can stay under the SEV threshold and still exhaust a budget.

Internal objectives for every method on every service will create a wall of red that nobody reads. Choose the journeys whose failure is the business: search, enroll, capture, sign-in. Give everything else a coarser objective or none, and say so. An unmeasured service is an explicit residual, not an accident, when you wrote the sentence.

Architecture and Design Diagrams

The path from a scenario to a release decision:

mermaidCopy
flowchart TD
    scen[Six-part scenario] --> sli[SLI: good events over valid events]
    load[Workload envelope from capacity model] --> slo[SLO: target plus window plus load]
    sli --> slo
    slo --> budget[Error budget = 1 - target]
    budget --> rec[Recording rule retained for the window]
    rec --> burn[Burn-rate alert]
    burn --> policy{Budget state}
    policy -->|Healthy| ship[Feature releases allowed]
    policy -->|Fast burn| stop[Stop the bleed, hold risk]
    policy -->|Exhausted| pay[Pause risky releases and repair]

Where the two example SLIs are collected. Search is measured at the gateway, including sheds. Capture is measured at the payment boundary, after reconciliation, not at the pod health check.

mermaidCopy
flowchart LR
    learner[Learner] --> gw[Gateway]
    gw --> search[Search API]
    gw --> enroll[Enrollment API]
    enroll --> pay[Payment service]
    pay --> acq[Acquirer]
    gw --> searchSli[Search SLI: status and latency]
    pay --> paySli[Capture SLI: terminal reconciled outcome]
    search --> idx[(Search index)]
    pay --> ledger[(Payment ledger)]

A dependency product is an input to design, not the published SLO:

mermaidCopy
flowchart TD
    a[Search 99.9%] --> p[Product under independence]
    b[Enroll 99.9%] --> p
    c[Capture 99.95%] --> p
    p --> note[Labeled assumption only]
    edge[Measured journey at the gateway] --> published[Published user SLO]

Technical Explanation

Good events, valid events, and exclusions

Write the SLI as code would write a predicate. For search:

• Valid: request reached the gateway, passed authentication, method and path match the search route, body within the size limit.
• Good: valid, and the gateway status is 2xx, and the gateway duration is ≤ 300 ms.
• Shed, counted neither good nor bad when the arrival rate is above the documented line and the status is 429 with Retry-After. Below that line, 429 is a bug and counts as bad.
• Excluded: 401 and 403 from authentication, and client-closed connections before 50 ms that never reached the service. Exclusions are listed so an incident cannot invent a new one.

For capture:

• Valid: a capture attempt with an idempotency key, after the learner submitted payment.
• Good: the attempt reaches a terminal state CAPTURED or DECLINED that matches the acquirer, within 15 seconds, and a repeated key does not create a second capture.
• Bad: timeout with unknown outcome still unresolved after 15 seconds, a 5xx, a double capture, or a 200 that later reconciles differently.
• Explicitly bad to shed: dropping a capture to protect CPU. Search may shed. Capture queues or returns a retriable error with the same idempotency key. Load shedding from lesson 35 is path-specific.

These predicates are the design. The percentage is a negotiation on top of them. Teams that negotiate the percentage first end up measuring whatever the gateway already counts.

Histograms, windows, and recording

A Prometheus histogram bucket is cumulative: le="0.3" includes every request faster than 0.3 seconds. Subtract adjacent buckets if you want a distribution for a human. For the SLO you want the cumulative ratio, so do not subtract.

histogram_quantile estimates a percentile by interpolation inside a bucket. It is the right tool for a graph and a noisy tool for a page. If your SLO is "99 percent under 300 ms," query the le="0.3" ratio directly so you are not paging on an interpolation artifact. Keep histogram_quantile for the performance review in lesson 65, where a human is looking at the shape.

Recording rules should store sli_good and sli_valid as counters' rates aggregated to one minute, summed, and retained. The 30-day ratio is then a sum of those recordings. Computing rate(...[30d]) on raw high-cardinality series at query time will either time out or tempt someone to shrink the window until the dashboard is green. Cardinality is part of the SLI design: uri as a template, not the raw path with ids; no learner id in the metric; status class rather than a free-form error string if you do not need the string for the ratio.

OpenTelemetry histograms can feed the same ratios. Lesson 43 chooses the instrumentation path. This lesson only insists that whatever path you pick can answer the predicate for the full window. Two pipelines that disagree are a broken SLI, and the policy should say which one governs releases until you fix the other.

Burn rate, computed once

For a 30-day window the even burn consumes (1/720) of the budget per hour, because (30 \times 24 = 720). An observed error rate divided by the SLO's allowed error rate is the burn rate. Allowed error rate for 99.9 percent is 0.001. An observed 0.0144 failure ratio is a burn rate of (0.0144 / 0.001 = 14.4). Over one hour that consumes (14.4 / 720 \approx 2) percent of the monthly budget. The arithmetic is short on purpose. If the on-call cannot recompute it, the alert is a magic number and will be silenced.

Multi-window confirmation means the one-hour burn and the five-minute burn must both be above the threshold. The five-minute window is not a more aggressive SLO. It is a check that the bleed is still happening at the moment you would page. Put this sentence in the runbook or the alert will be "improved" into a single short window during an outage, and the team will stop trusting pages.

Component math without false precision

Three nines times three nines times three nines is 99.7 percent, to one decimal in the percentage. Reporting 99.7002997 percent pretends the inputs were exact. Lesson 24's rule holds: two significant figures and a labeled assumption. If capture's objective is tighter than search's, multiply the actual objectives you set, not a slogan of "three nines" pasted on all three.

Synchronous calls on the critical path multiply. Asynchronous work with an outbox, as in the enrollment decision of lesson 61, removes that component from the request SLO and creates a freshness SLO instead: 99 percent of receipts sent within fifteen minutes. That is a different SLI, with its own budget. Deleting a dependency from the request path is often the cheapest way to make a journey objective believable. The mail vendor's availability no longer has to be bought with a pool of HTTP threads.

Capacity, headroom, and the SLO as a spending cap

Meeting a latency SLO at a stated peak requires the saturated resource to sit below the knee from lesson 25. The SLO does not tell you the replica count. It tells the capacity review which peak it must pass, and it tells finance the cost of a tighter number. If search p99 only holds under 300 ms when you run three times the steady capacity, that factor is the price of the objective. Put the price next to the SLO when product asks for 99.99 percent or for 100 ms. Architecture Performance and Capacity Review is where the evidence is judged. Here you only refuse to adopt a number whose envelope was never costed.

Headroom is not an SLO. It is how you keep one. Running at 50 percent CPU with no latency objective is waste. Running at 95 percent utilization because the average looks fine will miss the tail. The policy can say: latency SLOs are evaluated at the documented peak, and a load test at that peak is a release gate for changes to the query path. The gate is the connection between this lesson and lesson 24's model.

Payment and search, side by side

The table is the antidote to the shared "three nines" slide. Search spends budget on purpose. Capture treats budget consumption as a defect to repair, and its tighter objective reflects money, not prestige.

Practical Implementation or Design Walkthrough

Problem statement

Replace the shared "three nines" sentence for TechLearningPro search and payment capture with SLIs, SLOs, budgets, and a policy. The systems already exist. You are not redesigning the acquirer integration. You are defining the numbers the next release is allowed to risk.

Assumptions and constraints

Assumptions, labeled: 30-day rolling window; search peak 1,500 rps during enrollment week and 200 rps on an ordinary evening; capture peak 30 rps; gateway is the user-facing clock for search; payment service is the system of record for capture outcome; Micrometer timers are already scraped as http_server_requests_seconds_*; raw retention is 15 days, so recording rules are required.

Constraints: card data stays at the acquirer, as the payment case study already decided; search may serve a stale index; capture may not drop requests to save CPU; the identity provider's failures are visible but are a separate SLO owned by that platform.

SLI implementation

Tag the timer with a route template. In a Spring Boot application the stock http.server.requests timer already records uri, status, and method when the observation API is enabled. Confirm the uri is the pattern /api/search and not /api/search/java-concurrency, or the cardinality will wreck the store and the ratio will be a sample of whatever series survived.

A recording rule, conceptual Prometheus, evaluated every minute:

promqlCopy
record: sli:search_valid:increase1m
expr: sum(increase(http_server_requests_seconds_count{uri="/api/search",status!~"401|403"}[1m]))

record: sli:search_good:increase1m
expr: sum(increase(http_server_requests_seconds_bucket{uri="/api/search",status=~"2..",le="0.3"}[1m]))

increase over one minute on a counter is the right input to a later sum over 30 days of recorded samples. Using rate and then multiplying back by time is a common way to drift. Keep units as events. The good rule as written counts fast 2xx responses. The valid rule excludes authentication failures. A 429 is valid and, under the policy, is good only when a companion gauge says the shed line was exceeded. That conjunction is easier in a recording script than in one expression. Write it in the rule file and test it with a fixture. A 429 that always counts as good will hide a broken limiter.

Capture does not use the HTTP histogram alone. The payment service increments a counter when reconciliation finishes:

javaCopy
public final class CaptureMetrics {
    private final MeterRegistry registry;

    public CaptureMetrics(MeterRegistry registry) {
        this.registry = registry;
    }

    public void record(String terminalState, boolean reconciled, Duration latency) {
        Timer.builder("payment.capture.latency")
            .tag("state", terminalState)
            .register(registry)
            .record(latency);
        Counter.builder("payment.capture.outcome")
            .tag("state", terminalState)
            .tag("reconciled", Boolean.toString(reconciled))
            .register(registry)
            .increment();
    }
}

terminalState is an enum you control: CAPTURED, DECLINED, UNKNOWN. It is not the acquirer's raw string. High-cardinality tags would make the SLI unqueryable, which is a silent way to lose the objective. The good event is reconciled=true and state in {CAPTURED, DECLINED} and latency under 15 seconds. UNKNOWN older than 15 seconds is bad even if a later job fixes it; the fix is a recovery, and the bad event still spent budget. That choice keeps the SLO aligned with user harm. If you only count the state after the nightly job, you will report 100 percent on a day of timeouts.

Registering a new counter on every call is acceptable in Micrometer because the registry caches the meter by name and tags. Do not put the idempotency key in a tag. Ever.

Budgets

Search availability, request-based, assumed (4 \times 10^8) valid requests in 30 days at a blended rate. The assumption is the lesson 24 envelope, not a measurement, until the recording rule has been live for a month. Budget at 99.9 percent: (4 \times 10^5) bad requests. At a peak of 1,500 rps, one bad minute is (1{,}500 \times 60 = 90{,}000) bad requests, about 22 percent of the monthly budget. That single number justifies a burn-rate page. It also justifies the shed line: beyond 1,500 rps you would rather exclude sheds than spend the month in the first hour of enrollment week.

Capture at 30 rps blended much lower, assume (3 \times 10^6) attempts in 30 days. At 99.95 percent the budget is 1,500 bad attempts. One bad minute at 30 rps is 1,800 attempts, which is the entire month. The latency allowance of 15 seconds and the refusal to shed are how you avoid spending it on a blip. The tighter percentage is honest only because the attempt volume is small and the team can look at every bad attempt. At search's volume, 99.95 percent would be an alert factory unless the path is genuinely that calm.

Policy text you can adopt

1. Search feature releases continue while the rolling budget is under 50 percent consumed and no page-level burn is firing.
2. A page-level burn freezes search experiments, including relevance model pushes, until the burn ends and a human acknowledges residual risk.
3. Capture budget consumption above 10 percent in a week freezes all payment-path changes except fixes for the consuming defect. There is no "experiment" exception.
4. A 429 on capture is a SEV, regardless of the search policy.
5. Vendor acquirer failures count against the capture SLO. The action may be contractual. The graph does not get a filter during the review.

Validation

• A unit fixture feeds a histogram and checks that a 301 ms request does not increment the le="0.3" good count.
• A staging test runs searches at 200 rps and at 1,500 rps and records the slow fraction. The SLO is not "passed" by one run. The run calibrates the envelope.
• A game day delays the acquirer by 20 seconds and expects UNKNOWN to count as bad until reconciliation completes, then expects the original bad minute to remain in the 30-day sum. If the dashboard goes green when the job finishes, the recording rule is wrong.
• Delete a label with an id in it from any metric used in the SLI, and fail CI if it returns. Cardinality is a correctness property of the indicator.

Expected result

Product can read one table and see that search and capture are not the same promise. On-call has a burn-rate query they can recompute. Releases have a written response to a spent budget. The old slide with "three nines" for both systems is superseded by the ADR that links this table, in the record form from lesson 10.

Enterprise Real-World Scenarios

The search objective that was copied from payments

Search inherited 99.99 percent and a 100 ms p99 from a platform template. The index was a replica of catalog updated by a job every minute, which lesson 27's invalidation discussion would already have called a freshness choice. Meeting 100 ms at enrollment peak required a second in-memory cluster and a ban on relevance experiments, because any experiment moved the tail. The team spent a quarter chasing a number nobody had tied to a learner complaint. Session recordings showed learners tolerating a few hundred milliseconds, and abandoning only when the page spun past a second or returned an error.

The staff rewrite set the latency SLO at 99 percent under 300 ms, the availability SLO at 99.9 percent, and a separate freshness SLI at sixty seconds of index lag. The in-memory cluster proposal was withdrawn. The budget it would have consumed went to a shed line at 1,500 rps and a recording rule that had been missing, which meant the old 99.99 percent figure had never been measured over 30 days in the first place. It had been a graph of a seven-day p99, averaged across instances. Averaging those percentiles had hidden two nodes that were slow whenever a particular shard was hot.

The incident worth remembering happened after the rewrite. A relevance experiment burned 30 percent of the search budget in a day by doubling tail latency, not by returning errors. The burn-rate page fired. The policy froze experiments. The rollback took twenty minutes. Under the old regime the same experiment would have been "within SLA" because no contract existed and the averaged p99 moved only slightly. The SLO did its job by being loose enough to be real and tight enough that a bad experiment was visible in the currency the team had agreed to spend.

Capture looked up while learners were charged once and entitled never

Payment's dashboard was a process health check and a count of HTTP 200s from the capture handler. During an acquirer brownout the handler returned 200 after writing UNKNOWN, and a worker was supposed to resolve the unknown. The worker's queue grew for forty minutes. Learners refreshed, posted again, and the idempotency key saved them from a second charge, which is why the day did not become a financial incident. It became an access incident: entitlements were granted only on CAPTURED, so paid-but-unknown learners had no course until the worker drained.

The request-success SLI stayed at 100 percent. Support counted hundreds of tickets. The repair was the capture predicate in this lesson: UNKNOWN past 15 seconds is a bad event, the journey SLI for "paid and entitled" sits at the edge of those two facts, and a health check is not an input. The team also set the objective at 99.95 percent of attempts, request-based, and calculated that forty minutes at peak would have been many budgets over. That arithmetic got the worker its own latency SLO and a page on queue age, which is the same shape as the outbox age in lesson 61. The shared library that had been allowed to shed "all downstream calls" lost its permission to wrap the capture client. Search kept shedding. The code split followed the SLO split, which is the order you want. A shared resilience library that does not know which SLO it is protecting will spend the wrong budget.

Best Practices

• Start from the journey the learner can name. Add component SLOs only where a team boundary needs a local target.
• Write the good-event predicate before the percentage. Include exclusions on the same page.
• Choose request-based or time-based availability explicitly. Quote minutes only for the definition you implemented.
• Give latency an error budget by counting the fraction inside a threshold. Keep percentile charts for humans.
• Put the workload envelope and the shed line in the SLO text. A number without a load is not comparable across quarters.
• Store good and valid as recorded counts retained for the full window. Do not depend on raw high-cardinality series at 30 days.
• Set SLAs looser than SLOs, or do not set an SLA. Never let a contract number be tighter than the internal target.
• Give search and payment different shed policies. Write the exception in the library's allowlist.
• Page on burn rate with a short confirmation window. Ticket on slow burns. Do not page on a single host down if the SLO is a user ratio.
• When the budget is gone, change the release rules. An SLO with no policy is a graph.
• Recount the dependency product when the critical path changes. Removing a synchronous vendor call is often the SLO design.
• Reopen the objective when the measured peak exceeds the envelope, using the decision practice in lesson 61.

Common Mistakes and Anti-Patterns

The copied nine. Every service on a slide shares one percentage. It happens because a template feels consistent. The consequence is search over-built and capture under-specified, or the reverse. The repair is a predicate per journey and a percentage that matches harm and volume.

Uptime of the process. The SLI reads a health endpoint or a process restart count. It happens because the metric already exists. The consequence is a green dashboard during UNKNOWN payments and during error responses that returned quickly. The repair is a good-event definition at the user or ledger boundary.

Averaged percentiles. Per-instance or per-minute p99 values are averaged into a number that matches no set of requests. It happens because the query is easy. The consequence is hidden hot shards and false comfort. The repair is a histogram ratio over the aggregated events, weighted by traffic.

SLO tighter than the design. 99.99 percent is adopted with a single zone and a human failover. The budget is 4.32 minutes and the failover has never been practiced. The repair is to set the objective the current fault model can meet, and to track a separate prospective objective for the DR project. Lesson 64 is where that project has to fit the minutes.

Exclusions added during the incident. Client errors, then vendor errors, then "that deploy," are removed until the ratio is green. It happens because the review rewards the graph. The consequence is an SLO that no longer measures harm. The repair is a written exclusion list and a policy that vendor faults still count.

Shed-by-default. A shared bulkhead drops whatever is slow, including capture. It happens because lesson 35's tactic was adopted without the predicate. The consequence is lost payments that look like successful protection. The repair is an allowlist of routes that may return 429, and a capture path that keeps the idempotency key and retries.

Window theater. The objective says 30 days and the graph shows 24 hours because retention is short. It happens because nobody costed the recording rule. The consequence is a monthly review of a daily mood. The repair is recorded good and valid counts with a retention longer than the window.

Troubleshooting and Design Review

Symptom: The SLO dashboard is green and support volume is high.

Likely cause: the SLI counts the wrong good event, such as HTTP 200 or process up, or it excludes the path users are on.

Investigation: read the predicate. Compare a ticket's timestamp to the series. Check for filters added "temporarily."

Corrective action: redefine good events to match the ticket's harm. Backfill nothing; start the window honestly and note the gap.

Symptom: Pages fire every evening and are ignored.

Likely cause: the alert is on a short-window error ratio with no burn-rate context, or the threshold is the SLO itself with no room.

Investigation: compute the burn rate for the last page. See whether a five-minute confirmation window would have suppressed it. Check traffic; a low-traffic night makes ratios swing.

Corrective action: move to multi-window burn rates. For low-traffic capture, add a minimum attempt count before a ratio pages, and page on absolute bad events when the count is small.

Symptom: Search latency SLO fails only on enrollment week.

Likely cause: the objective has no envelope, and the test load was the ordinary evening.

Investigation: compare the week's rps to the lesson 24 table. Inspect shed behavior and the slow fraction below the shed line.

Corrective action: either hold the SLO only under the stated peak and shed above it, or fund the capacity that keeps 300 ms at the higher rate. Record the choice.

Symptom: Journey availability is far below the product of component SLOs, or far above it, and nobody trusts either number.

Likely cause: dependence. A shared failure made the product pessimistic or a measured journey includes retries the components count as failures.

Investigation: pick one incident and see which components failed together. Look for client retries multiplying bad events.

Corrective action: publish the edge measurement as the user SLO. Keep the product as a design estimate with the independence label. Fix double counting before you chase nines.

Symptom: Capture SLO requires 15-second terminal states, and the acquirer's documented timeout is 30 seconds.

Likely cause: the objective was chosen from a product sentence without the dependency's contract.

Investigation: read the acquirer timeout and the unknown-outcome path side by side.

Corrective action: set the SLI window to something the acquirer can miss without forcing a false DECLINED. Keep the user-visible "we are still confirming" state inside that window so the UI matches the predicate.

Design-review questions: What is the good event, in one sentence a support agent would recognize? What is excluded, and who may add an exclusion? Is the window backed by retained recordings? What load is the latency number valid at, and what happens above it? Is this time-based or request-based? What is the budget in minutes or in events? Which journeys may shed, and which must not? If three dependencies stay on the path, what journey number does independence imply, and did you label it? What changes in the release process when the budget is half gone?

Interview Questions and Answers

Q1. What is the difference between an SLI, an SLO, and an SLA?

An SLI is the measurement: a defined ratio of good events to valid events, such as the fraction of capture attempts that reach a reconciled terminal state. An SLO is a target on that measurement over a window, such as 99.95 percent over a rolling 30 days, at a stated load. An SLA is a customer agreement with a remedy if a looser number is missed. I set the SLO tighter than any SLA so ordinary budget burn does not automatically create a credit. Many internal services have SLOs and no SLA, which is coherent. Using the three words interchangeably is how a sales target becomes an on-call page with no query behind it.

Q2. How do you compute an error budget for a 99.9 percent availability SLO?

The budget is 0.1 percent of the window. Over 30 days that is (0.001 \times 30 \times 24 \times 60 = 43.2) minutes if, and only if, the SLI is time-based. If the SLI is request-based, the budget is 0.1 percent of valid requests in that window. At (10^7) requests it is 10,000 failures. I state which definition the dashboard uses. A one-minute peak outage and a one-minute overnight outage are equal in the time-based budget and very different in the request-based budget. Payment should prefer requests. A "reachable every minute" promise should prefer time.

Q3. Why do you dislike alerting on average latency or on an averaged p99?

The average hides the tail a learner feels, and the average of per-minute or per-instance p99s is not the p99 of the combined requests. A quiet minute with a handful of slow calls looks disastrous and represents almost nobody; a busy minute can burn the real budget while the averaged chart stays calm. I define the latency SLI as the fraction of valid requests at or under a threshold, which for one unaggregated set is the same claim as a percentile, and I alert on how fast that fraction consumes its error budget. I still look at percentile charts in a performance review. I do not page on their average.

Q4. How should search and payment SLOs differ?

Search can return a slightly stale index, can shed with 429 above a documented arrival rate, and can spend error budget on relevance experiments. A sensible shape is 99.9 percent availability and 99 percent of requests under 300 ms, plus a separate freshness SLI. Capture counts reconciled terminal outcomes, not HTTP 200s, targets something tighter such as 99.95 percent because the volume is lower and the harm is a charge, refuses to shed, and treats UNKNOWN past the deadline as budget spent. Copying one sentence onto both runbooks is how a shared bulkhead drops payments.

Q5. Three services on the path each promise 99.9 percent. What do you tell the user?

Under an independence assumption the journey is (0.999^3 \approx 99.7) percent, or about 130 minutes in a 30-day time-based reading. I label that product as a design estimate. Shared zones and shared deploys correlate failures, so I do not publish the product as the user SLO. I measure the journey at the gateway. I also do not multiply objectives and pretend I multiplied achieved SLIs. If I need the journey number to be 99.9, I have to take a dependency off the synchronous path or give the components tighter objectives and pay for them.

Q6. What is a burn rate, and why is a short window involved?

Burn rate is the observed error rate divided by the error rate the SLO allows. For 99.9 percent, a 1.44 percent error rate is a burn of 14.4, which consumes about 2 percent of a 30-day budget in one hour. The SRE Workbook pairs that long window with a short confirmation window so you page only when the bleed is still going on, and it uses a slower burn, classically 6 over six hours, for a larger slice of budget. I do not wait for the month to end, and I do not page on a five-minute ratio alone. Low-traffic services also need a minimum event count, or the ratio is noise.

Q7. How do you connect an SLO to capacity planning?

I write the load next to the target: 300 ms at or below 1,500 searches per second, with shedding above that line. The load comes from the lesson 24 envelope and gets replaced by measurement when the recording rule has history. If the peak grows past the envelope, I reopen the decision: more capacity, a looser SLO, or a lower shed line. A latency number with no rps is how teams pass a load test on an empty afternoon and miss enrollment week. Lesson 65 reviews the evidence; the SLO is the bar that evidence has to clear.

Q8. What is the difference between availability and durability objectives?

Availability is about the answer the user just received. Durability is about whether an acknowledged write survives a stated fault. A database can be available and still have lost commits if it acknowledged them before replication. I write durability against a fault model: one zone, with synchronous commit as lesson 29 defines it. I do not paste an object-storage "eleven nines" figure onto a ledger. RPO in the disaster-recovery lesson is the wider rewind you accept when the fault model is exceeded. The SLO stays with the fault you intend to survive without data loss.

Q9. How do retries affect the SLI?

Client retries can turn one user failure into several bad events, or they can hide a failure if you only count the final success. I define the event at a stated boundary. For capture, the idempotency key makes retries one attempt logically, and a second terminal capture is a bad event by itself. For search, a user hammering refresh counts as multiple requests, which is what the gateway saw; the journey SLI can instead count sessions if product cares about "could they search at all in this minute." I pick one and document it. I do not let a retry library inside the service mark the call good because an inner retry eventually worked, if the user has already given up.

Q10. What do you do when the error budget is exhausted?

I follow a policy written in advance. For capture, that means freezing payment-path changes except the fix for whatever burned the budget. For search, it means stopping experiments and shipping reliability work. The freeze has an owner and an end condition, so it does not become folklore. I do not relax the SLO during the review to recover budget; that is a new decision with a stakeholder, done in the open, usually after the incident. I also do not exclude the ugly traffic to repaint the graph. If the objective was unrealistic, supersede it with the evidence attached.

Q11. A product manager asks for five nines on search. What do you say?

I translate it into about 26 seconds of downtime in 30 days, or a few hundred bad requests given our volume, and I ask which fault model produces that: multi-region active serving, no human in the failover, a tested shed, and a budget for the capacity to stay far from the latency knee. I show the current achieved SLI and the cost of the last nine we already bought. If the learner complaint is actually "search feels slow on Monday," the right objective is a latency fraction at peak, not a fifth nine. I will support five nines when the DR and capacity designs can fit inside 26 seconds. I will not type it into a slide first.

Hands-On Exercises

Exercise: SLOs for payment and product search

Problem statement. You are the owner of TechLearningPro's learner-facing reliability objectives for one quarter. Search and payment capture currently share the words "highly available." Produce the SLI predicates, the SLOs, the budgets, the recording approach, and the release policy. Use assumptions you label. Do not copy one percentage onto both services unless your predicates justify it.

Tasks.

1. Write a six-part scenario for one search journey and one capture journey. The response measure must be collectable.
2. Define valid events, good events, and exclusions for each. State whether a 429 is good, bad, or excluded, and under what condition.
3. Choose request-based or time-based availability for each, and compute the 30-day budget in events or minutes. Show the arithmetic.
4. Add a latency SLO that is a fraction under a threshold, and name the histogram bucket that implements it.
5. State the workload envelope and what happens when arrival rate exceeds it.
6. Estimate the enroll-plus-pay journey under an independence assumption, label it, and say where you will measure the real journey.
7. Write five lines of error-budget policy, including what freezes.
8. Note one durability sentence for capture and one freshness sentence for search, separate from availability.

Expected deliverables. A two-page SLO document, the budget arithmetic, one example PromQL or equivalent ratio, and a policy paragraph.

Evaluation rubric.

A document that says "99.9 percent uptime" for both systems, with no predicate and no window, is not an SLO design.

Lesson Summary

Non-functional requirements become operable when a six-part scenario grows an SLI, an SLO, and a budget with a policy. The SLI is a predicate of good events over valid events, measured on the artifact the user or the ledger actually experiences. The SLO adds a target, a window, and a load. The SLA, if it exists at all, sits looser than the SLO and carries a remedy. The budget is arithmetic: 43.2 minutes for a time-based 99.9 percent over 30 days, or 0.1 percent of requests, and you say which one you mean.

Search and capture do not share a sentence. Search can be slightly stale, can shed above a line, and can spend budget on experiments. Capture counts reconciled terminal outcomes, does not shed, and treats lingering UNKNOWN as consumption of a small budget. Dependency products are labeled estimates under independence. Journey SLIs are measured at the edge. Burn-rate alerts spend the budget in public, and a spent budget changes what the team is allowed to ship.

Architecture Security and Threat Modeling takes the next review lens. An SLO that stays green while a second tenant can read the first tenant's results is not a successful service level. It is an indicator pointed at the wrong harm.

Further Reading

• Beyer, Jones, Petoff, and Murphy, Site Reliability Engineering — the SLI, SLO, SLA, and error-budget chapters this lesson treats as the base vocabulary.
• Beyer, Murphy, Rensin, Kawahara, and Thorne, The Site Reliability Workbook — implementing SLOs and multi-window, multi-burn-rate alerts, including the 14.4 and 6 burn-rate examples.
• Bass, Clements, and Kazman, Software Architecture in Practice — scenarios and response measures, which remain the architectural form of an NFR.
• Google's SRE guidance on alerting on SLOs — burn rate as observed error rate divided by allowed error rate.
• Prometheus histogram documentation — cumulative buckets and why histogram_quantile interpolates. Use the bucket ratio when the SLO is a threshold fraction.
• Micrometer's Timer and Spring Boot's HTTP observations — the http.server.requests series used in the walkthrough, with route templates rather than raw paths.
• This course: Architectural Thinking and Quality Attributes, Capacity Estimation and Workload Modeling, Scalability and Performance Fundamentals, Observability: Logging, Metrics and Tracing, Rate Limiting, Throttling and Load Shedding, Design a Payment Processing System, Idempotency and Exactly-Once Processing.