Observability: Logging, Metrics and Tracing

Advanced System Design & Software Architecture · Lesson 43 · Microservices and Enterprise Architecture

Lesson Overview

A checkout that used to be one Spring Boot process is now a gateway, an order service, a payment service, and a Kafka consumer that starts fulfillment. A learner reports that the purchase spinner never finished. The order row exists with status AUTHORIZED. Payment’s on-call says their handler returned 201. Fulfillment’s lag is near zero. Four logs, four clocks, and nobody can name the request.

That gap is the difference between monitoring a process and observing a system. Monitoring tells you a threshold you already named has been crossed: CPU, a 5xx counter, a disk. Observability is the property that lets you explain a failure you did not anticipate, from the signals the system emits. In a microservice architecture the signals have to cross process boundaries or the explanation stops at the first hop.

Three signals carry different questions. A log is a timestamped record of a discrete event: “order ord-1842 authorized, learner L-19, amount 49.00 USD.” A metric is a number aggregated over time: authorizations per second, the 99th percentile of handler latency, the depth of the Hikari pool. A trace is a tree of spans for one request: the gateway span, the order span, the payment client span, and the consumer span that continues the same trace through a Kafka header. Logs explain one case. Metrics say whether it is rare. Traces show which hop.

This lesson wires those signals for TechLearningPro checkout with interfaces that exist: Micrometer’s MeterRegistry, Timer, and Observation; Spring Boot’s Micrometer Tracing bridge (micrometer-tracing-bridge-otel) and the W3C traceparent header; SLF4J’s MDC keys traceId and spanId; the OpenTelemetry Collector as the export hop; Prometheus for the time series. Service-level indicators are defined on the checkout the learner sees, not on a host that happens to be busy. Cardinality is treated as a production incident waiting in a label.

The same signals are what Module 5’s case studies will be graded on. A URL shortener, a payment processor, a notification fan-out, and a ride-dispatch loop are un-debuggable if the only evidence is “the pod restarted.” Circuit Breaker, Retry and Bulkhead Patterns tells you a dependency failed. This lesson is how you prove which dependency, for which learner, without turning the log pipeline into a second production database.

Learning Objectives

By the end of this lesson, you should be able to:

1. Separate a log event, a metric time series, and a trace span, and state which question each one can answer about a failed checkout.
2. Propagate W3C Trace Context (traceparent, and tracestate when a vendor needs it) from the gateway through HTTP and Kafka so one learner request stays one trace.
3. Put traceId and spanId on every structured log line through SLF4J MDC, and keep a separate support correlation id when a ticket must not depend on a sampled trace.
4. Record a Micrometer Timer or Observation with low-cardinality tags only, and explain why orderId or learnerId as a Prometheus label will exhaust the store.
5. Define a service-level indicator for checkout availability and latency, and distinguish it from a saturation metric on the CPU or the JDBC pool (RED versus USE).
6. Choose head-based probability sampling in the process (management.tracing.sampling.probability) versus tail sampling in the OpenTelemetry Collector, and say which errors each one drops.
7. Export traces and metrics through a collector rather than from every pod straight to the vendor, and name what you must not put in span attributes or log fields.
8. Read an exemplar or a shared trace id from a slow histogram bucket back to the spans, and hand that habit to the Module 5 designs where the product promise is a latency number.

Prerequisites

• Lesson 34, API Gateway and Backend for Frontend: the gateway is where a public request should start a trace. This lesson does not redesign routing.
• Lesson 33, Message Queues and Event Streaming: fulfillment continues on Kafka. A trace that dies at KafkaTemplate.send is an incomplete story.
• Lesson 41, Circuit Breaker, Retry and Bulkhead Patterns: retries create extra spans and inflate rate. You need the signals before you tune the breaker.
• Lesson 24, Capacity Estimation and Workload Modeling: an SLI without a traffic assumption is a slogan.
• Comfort with SLF4J, Spring Boot Actuator, and the idea of a Prometheus label. You do not need to have operated a collector cluster.

Detailed Core Concepts

What an event, a measurement, and a span each refuse to do

A log line is written when something discrete happens. It can carry the order id, the decision (AUTHORIZED or REJECTED), and a stable error code. It cannot, cheaply, answer “what fraction of checkouts were slower than 300 ms in the last hour.” Scanning terabytes of JSON to compute a percentile is how teams discover that their log vendor bill exceeded the feature it was debugging. Logs are for cases. Keep them structured, leveled, and free of payloads you would not read aloud.

A metric is already aggregated, or it is a histogram the scraper can aggregate. Prometheus scrapes a text exposition from /actuator/prometheus. Micrometer converts a Timer named checkout.authorize into a counter of events plus a cumulative histogram, conventionally checkout_authorize_seconds_count, _sum, and _bucket. The scrape is cheap because the labels were chosen when the meter was registered. A metric cannot reconstruct learner L-19’s spinner. If you add orderId as a tag to recover that case, you have built a log with the worst storage model in the building: a new time series per order, kept for the retention of the metrics cluster, indexed for queries you will never write twice.

A trace is a sampled tree. The trace id is 16 bytes, hex-encoded to 32 characters in the W3C header. Each span has its own span id (8 bytes, 16 hex chars), a parent span id, a name, a start, a duration, a kind (SERVER, CLIENT, PRODUCER, CONSUMER, INTERNAL), a status, and attributes. The tree answers “which hop.” It does not answer “how often,” because production sampling keeps a fraction of traces. A missing trace is not evidence the request was healthy. The metric answers that.

The three signals meet at the trace id. Micrometer Tracing copies traceId and spanId into the SLF4J MDC for the current span, so a log line and a span share a key. Prometheus exemplars, when the client and the scraper support them, attach a trace id to a histogram sample so a slow bucket can open the trace that fell in it. Do not hope a timestamp match will survive clock skew and a retry.

W3C Trace Context is the contract between processes

The header is traceparent. One legal value looks like this:

textCopy
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Four fields, dash-separated: version 00, a 32-hex trace id, a 16-hex parent span id, and flags. The least significant bit of the flags, 01, means the caller sampled this trace and the callee should record it. 00 means the caller did not, and the callee should still continue the same trace id if it propagates context, but it need not export a span. Inventing a private X-Trace-Id that every team formats differently is how a gateway span and an order span become two traces that share only a human’s guess.

tracestate is the vendor extension the W3C spec allows beside traceparent. Leave it to the propagator. Do not parse it in business code.

Baggage (baggage header, W3C Baggage) is key-value context that rides along for application use: a tenant id, a canary cohort, a test-traffic marker. It is not a trace. It is copied to every downstream call. A learner email, a card token, or an authorization decision does not belong there. Baggage is logged by accident more often than spans are, because people print “the headers.” The JVM bridge is Micrometer Tracing’s OpenTelemetry bridge (micrometer-tracing-bridge-otel) plus an OTLP exporter. HTTP and Spring Kafka instrumentation inject and extract traceparent. Application code adds a span only where that instrumentation cannot see a business step.

Correlation ids are for humans; trace ids are for the sampler

Support will paste whatever the learner’s receipt shows. That should be a stable request id or the order id, generated or accepted at the gateway and returned in the response. A common header is X-Request-Id. Treat a client-supplied value as untrusted: cap the length, reject newlines, and do not let it override traceparent. If the header is absent, the gateway mints one.

The request id goes on every log line and, if you need it on the trace, as a span attribute with a bounded name such as tlp.request_id. It is not a Prometheus label. The trace id may be absent from the vendor UI when the trace was not sampled. The request id is still in the log index, which you retain for the support window (often days, not the full metrics retention, and not forever). Teach support to search logs by request id or order id, then jump to the trace only when traceId is present and the backend stored that trace.

Low cardinality is the metric design constraint

A Prometheus time series is identified by the metric name plus its label set. checkout_authorize_seconds_count{result="authorized",region="ap-south-1"} is one series. Adding learnerId multiplies the series by the number of learners who checked out during scrape lifetime. A catalog of a few hundred thousand learners, a fifteen-second scrape, and a histogram with a dozen buckets becomes millions of series. The collector falls over, or the bill does, and you lose the metric you added the label to save.

Legal labels for this checkout timer: result with a closed set (authorized, rejected, dependency_down), payment_method if the set is small and stable (card, upi, wallet), region if you operate a handful of regions. Illegal labels: order id, learner id, SKU, exception message, URL path with the id still in it, email domain scraped from user input.

Micrometer will record whatever tags you pass. The guard is MeterFilter.maximumAllowableTags(meterName, tagKey, maximum, overflowFilter), which stops accepting new values of one tag and can deny the rest. Use it as a backstop for http.server.requests’s uri tag. Spring MVC already prefers the route template (/orders/{orderId}) over the raw path when the request matched a mapping. Requests that miss every mapping used to show up as a unique uri. A filter that collapses unknown paths to UNKNOWN or REDIRECTION is the difference between a useful HTTP dashboard and a cardinality incident during a scan.

RED, USE, and the SLI the learner can feel

The RED method (rate, errors, duration) fits a request-scoped service. For checkout you want the rate of attempts, the fraction that failed for a reason the learner felt, and a latency histogram. The USE method (utilization, saturation, errors) fits a resource: CPU utilization, runnable threads waiting for a core, disk errors; Hikari pool utilization, threads waiting for a connection, connection failures. USE explains why RED moved. RED is what you page a product owner about. A host at 90% CPU with a healthy checkout SLI is capacity work for office hours. A checkout SLI burning its error budget at 20% CPU is a dependency or a lock, and the host graph will lie to you.

A service-level indicator is a measurement of user-visible behavior. Two SLIs are enough to start checkout:

• Availability. Successful authorizations divided by valid attempts. “Valid” is a product decision. A 400 because the client omitted sku is the client’s bug; many teams exclude 4xx from the availability SLI and keep a separate client-error rate so a broken app release is still visible. A 401 from a bad token is excluded. A timeout talking to payment, a 503, and an unhandled 500 are included. Write the exclusion list down. An SLI that moves when a scanner sends garbage will page humans for the wrong reason.
• Latency. The fraction of valid attempts faster than a threshold, not the average. An average hides the tail the spinner shows. A histogram does not. “99% of successful authorizations complete in under 300 ms, measured at the gateway” is an SLI. The 300 ms number is the beginning of an SLO, which Non-Functional Requirements and SLO Design will turn into an error budget. This lesson stops at a correct indicator and a recording rule you can alert on. Measure at the boundary the learner waits on. Payment can be fast while the gateway retries it three times, so payment’s internal timer is not the checkout SLI.

Sampling is a product decision about which failures you can see

Head sampling decides at the start of the trace, before you know the outcome. Spring Boot’s property management.tracing.sampling.probability is that coin flip (1.0 records everything, 0.1 records about one trace in ten, 0.0 records none). The decision is written into the traceparent flags so the rest of the tree agrees. Head sampling is simple and bounded. It will drop most error traces when errors are rare. At one percent errors and a ten percent sample, most failures have logs and metrics and no trace. That is acceptable only if logs are sufficient to localize, which across four services they often are not.

Tail sampling decides after the trace has finished, in the OpenTelemetry Collector’s tail_sampling processor. A typical policy keeps every trace whose status is error, keeps every trace slower than a threshold, and keeps a small probability of the rest. The collector must see the whole trace, which means every service exports to a collector that can buffer, and the decision waits until the trace is complete. Tail sampling costs memory and adds a delay before a trace is searchable. It is the right default once checkout’s error rate is too low for head sampling to catch failures and the log line is not enough.

Do not sample independently in each service. A gateway that keeps the trace and a payment service that flips its own coin produces broken trees. Parent-based sampling, which the W3C flag encodes, is the rule: the child honors the parent.

Force a sample for a synthetic probe or a support session by starting the trace as sampled at the gateway. Do not give every learner a header they can set to 01 and use to flood the backend. If you honor incoming traceparent from the public internet, callers can mark every request sampled. At the edge, either strip inbound trace context and start a new trace, or accept it only from networks you trust. Internal hops must propagate. The public edge should not be a free sampling switch.

What crosses the collector, and what must not

Application processes export OTLP (OpenTelemetry Protocol) to a collector, usually HTTP on port 4318 at /v1/traces or gRPC on 4317. Spring Boot’s Micrometer OTLP registry uses management.otlp.tracing.endpoint (default http://localhost:4318/v1/traces). The collector forwards to your trace backend and can also scrape or receive metrics. Pods should not each hold a vendor API key and a retry buffer. The collector is the place for tail sampling, attribute stripping, and retry.

Span attributes are not a dumping ground. Put identifiers you will search (order.id, tlp.request_id) and low-cardinality enums (payment.method). Do not put the request body, the SQL text with bound learner data, the Authorization header, or the PAN. Prefer OpenTelemetry semantic conventions for new HTTP attributes (http.request.method, http.response.status_code). Micrometer’s existing HTTP meters still use method, uri, status, and outcome. Know which names your dashboard queries.

Logs follow the same redaction rules. A structured JSON line (Spring Boot 3.4’s logging.structured.format.console set to ecs or logstash, or a Logback encoder you already run) is indexable. A string you concatenated with the full Order object is a leak and a useless field. Log levels: INFO for a business transition with ids, WARN for a retryable dependency failure you expect, ERROR for a request the learner lost and a human may need. DEBUG stays off in production. An exception stack at ERROR on every retry is how an error budget and a log quota die together. Log the stack on the final failure, and a single warning per retry with the attempt number.

Architecture and Design Diagrams

The checkout path below is the one the signals have to cover. The gateway starts the trace. Order, payment, and the Kafka consumer join it. Metrics are scraped on a different clock from the trace export. Logs are shipped from stdout, which on Kubernetes means the node agent, not an in-process log socket, unless you have a reason to bypass the node.

mermaidCopy
flowchart LR
  Learner[Learner] --> GW[API gateway]
  GW --> Order[order-service]
  Order --> Pay[payment-service]
  Order --> Kafka[(Kafka topic order.authorized)]
  Kafka --> Ful[fulfillment consumer]

  GW --> Logs[stdout JSON logs]
  Order --> Logs
  Pay --> Logs
  Ful --> Logs
  Logs --> LogIdx[log index]

  GW --> Prom[/actuator/prometheus/]
  Order --> Prom
  Pay --> Prom
  Prom --> PromDB[Prometheus]

  GW --> Col[OTLP collector]
  Order --> Col
  Pay --> Col
  Ful --> Col
  Col --> TraceDB[trace backend]

A single sampled authorization, with the Kafka hop as a producer span and a consumer span linked by the same trace id. The support request id is an attribute, not a second trace.

mermaidCopy
sequenceDiagram
  participant L as Learner
  participant G as Gateway
  participant O as Order service
  participant P as Payment service
  participant K as Kafka
  participant F as Fulfillment

  L->>G: POST /checkout
  Note over G: start SERVER span<br/>mint X-Request-Id<br/>traceparent flags sampled
  G->>O: POST /orders<br/>traceparent + X-Request-Id
  Note over O: SERVER span child<br/>MDC traceId spanId
  O->>P: POST /charges<br/>traceparent
  P-->>O: 201
  O->>K: publish OrderAuthorized<br/>traceparent header
  O-->>G: 201
  G-->>L: 201 + X-Request-Id
  K->>F: deliver record
  Note over F: CONSUMER span<br/>same trace id

The SLI is computed from the gateway’s histogram, not from any internal timer. The recording rule is PromQL you would actually run. histogram_quantile needs the bucket series, and it estimates a quantile inside a bucket. It is not a sorted list of requests.

mermaidCopy
flowchart TD
  A[Gateway timer checkout.authorize] --> B[Prometheus scrape]
  B --> C["rate of _count by result"]
  B --> D["histogram_quantile on _bucket"]
  C --> E[Availability SLI]
  D --> F[Latency SLI]
  E --> G[Error budget burn]
  F --> G
  B --> H[Exemplar trace id on a slow bucket]
  H --> I[Trace backend]

Technical Explanation

Micrometer meters you can defend

MeterRegistry is the facade. A Counter only goes up: attempts, published events. A Timer records latency and a count; use it for any operation with a duration you might alert on. A Gauge reads a current level: pool active connections, consumer lag if you publish it as a gauge. Prefer a Timer for latency so the Prometheus name ends in _seconds.

Observation is the Micrometer API Spring Boot 3 instrumentation is built on. A timer plus a trace span plus an exemplar come from one observation if a tracing bridge is present. Creating a raw Timer and a raw OpenTelemetry span for the same business step double-counts latency and splits the story. Use Observation for your own business spans. Leave HTTP and Kafka to the instrumentation that already wraps them.

An observation tagged with a high-cardinality value is still a high-cardinality metric. lowCardinalityKeyValue is the method that becomes a label. highCardinalityKeyValue is documented as not becoming a metric tag; it can still become a span attribute. Use that split. orderId is high cardinality. result is low.

What the OTLP export does and does not guarantee

The exporter batches spans and retries briefly. A dead collector drops traces and must not fail the checkout. If export blocks the request thread, you have made the vendor part of the purchase path. Prometheus pulls /actuator/prometheus. A gauge supplier that calls the network, or a scrape that takes a lock the request path needs, goes blind during the incident. Keep the endpoint cheap.

Clocks and retries

Spans use a monotonic duration and a wall-clock timestamp. Skew between nodes can draw a child as starting before its parent, so trust durations more than a merged timeline. A retry is several client spans under one server span, and the learner waited for the sum. The gateway SLI must record that one outcome. Alerting only on payment’s 5xx hides an open circuit breaker: payment looks idle because the caller stopped sending. Page on the caller’s result=dependency_down, and keep consumer lag beside the checkout chart so a fast HTTP 201 that never unlocks the course is still visible. If unlock is asynchronous, publish a second indicator for unlock delay. The HTTP success is not the product promise.

Practical Implementation or Design Walkthrough

Problem

TechLearningPro’s checkout is four processes. Support must locate one learner’s attempt with the id printed on the receipt. On-call must know, within a minute, whether authorization success has fallen or the tail has blown past 300 ms. Engineers must open one trace for a sampled slow request and see gateway, order, payment, and fulfillment. The metrics store must survive a marketing email that sends half a million learners through checkout. Traces must not contain the card or the password. Export failure must not fail the purchase.

Requirements and constraints

• One trace id across HTTP and Kafka, W3C traceparent.
• Structured logs with traceId, spanId, tlp.request_id, and orderId. No PAN, no Authorization, no full body.
• A gateway timer checkout.authorize with tag result in a closed set.
• Availability SLI excludes 400 and 401. Latency SLI is the 99th percentile of successful authorizations at the gateway, target under 300 ms, reported as a fraction under the threshold as well as the quantile.
• Head sampling at 10% until the collector’s tail sampling is on; then the process sample rate goes to 1.0 into the collector and the collector keeps errors, slow traces, and 5% of the rest. Do not run both a 10% head sample and tail sampling, or the collector never sees the errors you wanted to keep.
• Prometheus labels bounded with MeterFilter.maximumAllowableTags on http.server.requests / uri.
• Java 21, Spring Boot 3.x, Micrometer Tracing OTEL bridge. Kubernetes stdout for logs.

The business observation

HTTP instrumentation already times the controller. The business result is not the HTTP status alone: a 200 with status=REJECTED because the card was declined is a successful system and a failed sale, and it must not page as an outage. Record the business outcome explicitly. highCardinalityKeyValue carries the order id onto the span. lowCardinalityKeyValue carries result onto the metric.

javaCopy
public final class CheckoutMeter {
    private final ObservationRegistry observations;

    public CheckoutMeter(ObservationRegistry observations) {
        this.observations = observations;
    }

    public Authorization authorize(OrderDraft draft, Supplier<Authorization> action) {
        Observation observation = Observation.createNotStarted("checkout.authorize", this.observations)
                .contextualName("checkout.authorize")
                .highCardinalityKeyValue("order.id", draft.orderId())
                .start();
        try (Observation.Scope ignored = observation.openScope()) {
            Authorization result = action.get();
            observation.lowCardinalityKeyValue("result", result.code());
            return result;
        } catch (PaymentUnavailableException ex) {
            observation.lowCardinalityKeyValue("result", "dependency_down");
            observation.error(ex);
            throw ex;
        } finally {
            observation.stop();
        }
    }
}

result.code() is an enum name the application controls: authorized, rejected, dependency_down. It is not ex.getMessage().

The backstop for route labels lives next to the registry. MeterFilter beans are applied by Spring Boot’s MeterRegistryCustomizer path when you declare them as beans.

javaCopy
@Bean
MeterFilter capHttpUriTags() {
    return MeterFilter.maximumAllowableTags(
            "http.server.requests",
            "uri",
            200,
            MeterFilter.deny());
}

Tracing properties that match the bridge

The process samples at the head only until tail sampling exists. The endpoint points at the collector, not at a vendor URL baked into the image.

yamlCopy
management:
  tracing:
    sampling:
      probability: 0.1
  otlp:
    tracing:
      endpoint: http://otel-collector.observability.svc:4318/v1/traces
logging:
  pattern:
    level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"

The pattern is the Spring Boot documented way to show Micrometer’s MDC keys in plain text logs. With structured logging, those MDC entries become JSON fields automatically if the encoder includes MDC. Add tlp.request_id yourself in a once-per-request filter that also echoes the response header. Generate it with a UUID when the inbound header is missing or longer than you allow. Strip CR and LF. Never copy it into a metric tag.

Kafka follows the same registry. Spring Kafka 3 creates observations around KafkaTemplate and the listener container when an ObservationRegistry is configured, which Spring Boot does when Micrometer is on the classpath. The tracing bridge puts traceparent on the record headers. The fulfillment listener then continues the trace. You do not invent a header name. You do confirm in one integration test that a produced record contains traceparent and that the consumer MDC shows the same traceId. That test is worth more than a dashboard screenshot.

SLI recording, stated as PromQL

Availability over five minutes, after the application has exported checkout_authorize_seconds_count (Micrometer’s Prometheus naming for the timer checkout.authorize):

textCopy
sum(rate(checkout_authorize_seconds_count{result=~"authorized|rejected"}[5m]))
/
sum(rate(checkout_authorize_seconds_count{result=~"authorized|rejected|dependency_down"}[5m]))

Declines (rejected) count as the system working. dependency_down counts against availability. This is a choice; document it next to the rule so the next on-call does not “fix” it during an incident.

Latency as a quantile of successful calls:

textCopy
histogram_quantile(
  0.99,
  sum by (le) (rate(checkout_authorize_seconds_bucket{result="authorized"}[5m]))
)

histogram_quantile is an estimate. Choose buckets that bracket 300 ms. Micrometer’s default percentile histogram can publish buckets; if you set SLO buckets yourself, publish only the ones you will alert on. A histogram with a bucket boundary at the SLO threshold makes the “fraction under 300 ms” query a simple ratio of two cumulative buckets, which is a better alert than a quantile estimate.

Failure handling inside the telemetry path

If the collector is down, checkouts still commit. The runbook says traces will be missing and logs plus metrics remain. If Prometheus cannot scrape, traces and logs remain, and someone should still be paged by a dead-man’s switch on the scrape itself: an alert that fires when the scrape fails, not only when an SLI dips. A silent metrics outage during a real outage is the second incident.

Redaction is a test, not a policy document. A unit test on the log encoder or a log-capture test asserts that a known PAN-shaped string and a sample Bearer token do not appear. Span attribute keys are an allow-list in the collector processor attributes with an action that deletes unexpected keys in non-production first, then in production once you trust the list.

Expected result

A learner receipt shows X-Request-Id. Support finds the log lines for gateway and order with the same traceId. If the trace was sampled, the UI shows payment’s client span and the fulfillment consumer span. The availability rule does not page on a wave of declines caused by a bank outage that payment correctly surfaces as rejected, and it does page when payment is unreachable. A load test that creates a million distinct order ids does not create a million Prometheus series.

Enterprise Real-World Scenarios

A bank-style ledger beside a learner checkout

A ledger posting and a learner checkout do not want the same sampling rate. Settlement volume is bounded and every lost posting trace is an audit argument. A marketing campaign is not. TechLearningPro checkout should tail-sample: keep errors, keep slow traces, and keep the capture route by collector policy, because a W3C trace is sampled or not as a whole from the parent flag. You cannot honestly force only the acquirer span to be stored if the gateway flipped the flag off. Financial state changes in Design a Payment Processing System are compliance logs with a retention window, and they are traces the sampler is biased to keep. They are still not Prometheus labels.

A notification fan-out that looks healthy and arrives late

Design a Notification Platform will accept a request, publish, and let workers deliver email and push. The HTTP SLI is green at 40 ms. Learners receive mail twenty minutes later because one provider is slow and lag is on a dashboard nobody owns. The indicator that matches the product is “fraction of accepted notifications delivered within two minutes,” measured from the accept timestamp on the message to the provider’s acceptance. That number is a metric recorded by the worker, joined to the originating trace by the traceparent on the Kafka record. The accept API’s latency stays a separate SLI so a slow HTTP handler is not confused with a slow provider. Two indicators, two owners, one trace. The same split applies to a ride platform’s “request accepted” versus “driver assigned,” and to e-commerce “order placed” versus “warehouse acknowledged.” Module 5 stops being a drawing exercise when each case study names the indicator at the boundary the user waits on and the indicator at the asynchronous completion.

Best Practices

• Start the trace at the edge you control, propagate W3C context on every outbound HTTP call and message publish, and test that propagation in CI.
• Define two SLIs at the learner boundary before you build a wall of USE graphs. Add pool and CPU charts as the explanation layer.
• Keep metric tags to closed sets. Put ids on logs and span attributes. Use MeterFilter.maximumAllowableTags so a bug cannot open the set.
• Separate business decline from dependency failure in the result tag, or your availability SLI will page the bank’s cardholders as if they were your outage.
• Sample with one policy for the whole trace. Prefer tail sampling for rare errors once a collector exists. Do not stack an aggressive head sample in front of it.
• Export OTLP to a collector. Keep vendor credentials and sampling policy out of the application image.
• Redact at the source and again at the collector. Test that a PAN and a bearer token cannot be logged.
• Log the final failure with a stack, and log retries as single warnings. Include traceId even when you also have X-Request-Id.
• Alert on SLI burn and on telemetry health (scrape failed, exporter error ratio, collector queue). A green board that is not updating is a failed board.
• During a canary, split the SLI by a low-cardinality version or cohort tag so the new pods can be compared. Drop the tag if the set of versions grows without bound; deployment version during a long rollback window should stay a handful of values.

Common Mistakes and Anti-Patterns

Logging the body “just this once.” The once becomes the encoder default. Card numbers and tokens land in a log index with a wide access list. The better path is an explicit allow-list of fields at INFO, and a DEBUG logger that cannot be enabled in production without a change ticket. Consequences are regulatory as much as operational.

Using the trace id as the only support key, with a 10% sample. Nine of ten tickets have no trace, and the logs do not carry another id, so support closes them as “cannot reproduce.” Keep X-Request-Id on every line regardless of sampling.

Tagging metrics with the order id to make them “more debuggable.” The store falls over on the day you most need it. Ids belong on spans and logs. Exemplars exist so a histogram bucket can point at a trace without a series per order.

Paging on averages and on causes. Average latency stays under 300 ms while the 99th sits at four seconds. CPU alerts wake people for a batch job the SLI never felt. Page on the learner SLI. Use cause graphs after the page.

Counting retries as extra failed requests at the gateway. The gateway SLI should record the outcome the learner received once. Internal client metrics may count each attempt, labeled attempt, so retry storms stay visible without burning the product error budget twice. Mixing them makes every incident argument about the denominator.

A private trace header per service. Order accepts X-B3-TraceId, payment accepts traceparent, fulfillment accepts a Kafka header named correlationId that is actually the order id. Nobody can jump hops. Standardize on W3C at the platform level and treat a second format as a migration with a dual-read at the edge, not as team preference.

Synchronous export on the request thread. A slow vendor becomes checkout latency and then checkout failure. Telemetry is asynchronous and lossy. The purchase is not.

Troubleshooting and Design Review

Symptom: logs in order and payment do not share a traceId, and payment has no traceparent in its access log. Likely cause: the outbound client is a raw HttpURLConnection or an uninstrumented client the bridge does not wrap, or a filter rebuilds the request and drops headers. Check: one captured downstream request in a test. Fix: use the Spring-instrumented RestClient or WebClient, or inject the Micrometer Propagator yourself on that one client. Do not start a new trace in payment and call it correlation.

Symptom: Prometheus memory climbs linearly with traffic, and the high series are http_server_requests_seconds with uri values that look like /orders/ord-1842. Likely cause: a catch-all mapping or a missing template. Check: count by (uri) is the query. Fix: template the route, deny unknown uris via MeterFilter, and restart scrape only after the series churn stops. Deleting the series without fixing the tag just delays the next climb.

Symptom: the latency SLO page fires, traces for the minute are all fast. Likely cause: head sampling kept the fast majority, or the dashboard quantile is on a different metric than the SLI. Check: compare histogram_quantile labels to the alert expression, including result. Fix: tail-sample slow traces, and put the alert expression in the dashboard title so they cannot drift.

Symptom: fulfillment spans exist but never share the producer’s trace id. Likely cause: a producer interceptor strips headers, or the consumer builds a new span with tracer.spanBuilder without making the extracted context the parent. Check: the raw Kafka header traceparent on a record in a test topic. Fix: leave Spring Kafka’s observation propagation in place; if you must create a manual span, extract the header with the configured Propagator and pass the extracted context as the parent.

Symptom: checkout returns 500 when the collector rolls. Likely cause: a blocking exporter or a log appender that throws on a full buffer. Check: thread dump of request threads stuck in export. Fix: non-blocking appenders, a dropping queue, exporter timeouts. Checkout’s success must not depend on the collector process.

Design review questions. Where is the SLI measured, and which status codes are excluded? What is the closed set of metric tags? Is inbound traceparent from the internet honored? Where are PAN and Authorization stripped? What remains debuggable when the trace backend is down for an hour? Which indicator covers work that happens after the HTTP response?

Interview Questions and Answers

Q1. What is the difference between a log, a metric, and a trace?

A log is a discrete event with a timestamp and fields, used to explain one case such as a single order id. A metric is a numeric series aggregated in advance, used to answer rates, error ratios, and latency distributions. A trace is a tree of timed spans that share a trace id, used to see which hop in one sampled request was slow or failed. Metrics are the only one of the three that stays cheap at full traffic, and only if labels stay low cardinality. Traces are sampled. Logs sit in between: you can keep one structured line per request more easily than you can keep a full trace, and you still cannot compute a percentile from them cheaply.

Q2. Why is orderId safe on a span and unsafe on a Prometheus label?

A span is stored once per sampled request. An attribute with the order id adds a few bytes to that document, and you search it when you already have the id. A Prometheus label becomes part of the series identity. Each distinct order id creates a new series for every histogram bucket and every other label combination, and those series live for the metrics retention window. Under a sale, that is unbounded growth in the metrics database. The join from a chart to a single order is an exemplar or a log line carrying the trace id, not a label.

Q3. What does the W3C traceparent header contain, and why do you not invent your own?

It contains a version, a 16-byte trace id, an 8-byte span id, and flags, in hex, dash-separated. The sampled flag tells the callee whether to record. Services that propagate this header, including through a Kafka record header, reconstruct one tree. A private header forces every team to write a propagator, loses interop with standard instrumentation, and splits one checkout into several traces. tracestate is for vendor-specific state. Baggage is a different header and is not the trace.

Q4. How do you keep a support ticket debuggable when only 10% of traces are stored?

You mint or accept a bounded X-Request-Id at the gateway, return it to the client, and put it on every log line and as a span attribute. Sampling does not drop logs. Support searches the log index by that id or by the order id. When the log line includes a traceId and the trace was sampled, they open the trace. The request id is mandatory. The trace is best-effort.

Q5. What is an SLI, and how is it different from a USE metric on the JDBC pool?

An SLI measures behavior the user can feel, at a boundary you name. Checkout availability as successful outcomes over valid attempts, and the fraction of those attempts under 300 ms at the gateway, are SLIs. Pool utilization, threads waiting for a connection, and connection errors are USE signals about a resource. They explain an SLI miss. They are the wrong page if the pool is hot and learners are still inside the threshold, and they never fire if the pool is idle because payment is the thing that is down.

Q6. Why can head-based sampling hide the outage you built tracing to see?

The process decides to record the trace before it knows whether the request will fail, usually with management.tracing.sampling.probability. Rare errors are mostly discarded. You still have the error metric, but not the tree that shows which hop failed. Tail sampling in the collector sees the finished trace and can keep all errors and slow calls, plus a fraction of the rest. Tail sampling needs the collector to receive complete traces, so the process must export more than the final keep rate, and it must not apply a second, aggressive coin flip first.

Q7. A retry calls payment three times. What should the gateway SLI count?

One learner outcome. If the third attempt succeeds and the learner gets 201, availability counts one success and the latency includes the time they waited. Client-side metrics on the payment call can count three attempts so the retry storm is visible. Counting three gateway failures for one user-visible success burns the error budget for behavior the user did not see as failure, and it trains the team to ignore the page. The span tree should still show three client spans so the time is explainable.

Q8. What should you refuse to put on a span or a log line?

Secrets and payment data: Authorization and Cookie headers, passwords, PANs, full bank responses, raw SQL with bound personal data, and the entire request body. Also refuse unbounded text such as exception messages used as metric tags. Identifiers you already show the learner, stable error codes, durations, and low-cardinality enums are appropriate. Redact in the application and drop unexpected attribute keys in the collector. A test that feeds a fake PAN through checkout and asserts it never appears in captured logs is part of the control.

Q9. Prometheus cannot scrape and the collector is down. What do you still have, and what did you do wrong if checkout is also down?

You should still have process stdout, which on Kubernetes the node agent can ship, and you should still complete purchases. Tracing and metrics are best-effort. If checkout returns 500, the exporter or the log appender is blocking or throwing on the request thread. Fix that coupling. Separately, alert on scrape failure and exporter errors so telemetry loss is its own page and not something you discover after the fact, when the graphs are suspiciously flat.

Q10. How does this design show up in a Module 5 case study such as notifications or payments?

You name the SLI at the boundary the user waits on, and a second SLI for asynchronous completion if success is not the HTTP response. You propagate traceparent onto the message so the worker span joins the accept span. You keep ids out of metric labels even when the case study tempts you to tag by userId or deviceId. You state the sampling policy and the redaction list next to the data flow. A diagram with boxes and no indicator is not an operable design. The payment case study biases retention toward capture traces and compliance logs. The notification case study refuses to call the accept API’s latency the delivery promise.

Hands-On Exercises

Exercise 1: Specify checkout observability

Problem. Produce the observation design for the four-hop checkout in this lesson: gateway, order, payment, Kafka fulfillment.

Tasks.

1. List the log fields on the authorization line and the fields that are forbidden. Include the support id and the trace id.
2. Define the gateway timer name, the closed result tag set, and one MeterFilter you would register.
3. Write the availability SLI in words and in PromQL, including what happens to declines and to payment timeouts.
4. State the latency SLI threshold and whether you will alert on histogram_quantile or on a bucket ratio. Justify the choice.
5. Decide head versus tail sampling for a 1% error rate at 200 requests per second, and say what the process exports.
6. Show where traceparent is created, where it is copied onto the Kafka record, and what a public client is allowed to send.
7. Name the dashboard panels that are RED and the panels that are USE, and which one pages.

Deliverables. A one-page signal contract, the PromQL, the sampling decision, and a sequence diagram equivalent to the one in this lesson.

Acceptance. A million distinct order ids cannot increase Prometheus series. A ticket with only X-Request-Id can be found when the trace was not sampled. A collector outage does not fail POST /checkout.

Exercise 2: Review a dashboard proposal

The proposal adds Prometheus labels learnerId, sku, and exceptionMessage to checkout.authorize, logs the full JSON body at INFO, samples each service at 50% independently, and pages when average CPU exceeds 70%. Write the review. For each item, state the failure mode and the replacement signal or control. Keep the critique specific to cardinality, sampling flags, redaction, and SLI placement.

Rubric.

Lesson Summary

Logs explain one order, metrics explain the rate and the tail, and traces explain one sampled path across processes. The join key is the W3C trace id, copied into SLF4J MDC as traceId and spanId, plus a support X-Request-Id that does not depend on sampling. Micrometer Observation and Timer feed Prometheus only with low-cardinality tags; order ids stay on spans. RED at the gateway is the checkout SLI, USE on pools and CPU explains it, and a second indicator is required when the learner’s real success happens after the HTTP response, on the other side of Kafka. Head sampling is a coin flip that drops rare errors. Tail sampling in the OpenTelemetry Collector can keep those errors if the process actually exports them. Telemetry export must be lossy and off the critical path. Those rules are the ones Module 5’s payment, notification, e-commerce, and dispatch designs have to state out loud, or the architecture cannot be operated.

Further Reading

• W3C Trace Context — traceparent, flags, and the rule that a trace id is not a private header.
• W3C Baggage — what may ride along, and why it is not a place for secrets.
• OpenTelemetry documentation — traces, OTLP, semantic conventions, and the collector’s tail sampling processor.
• Spring Boot, Tracing — management.tracing.sampling.probability, the OTLP endpoint, and MDC traceId / spanId.
• Micrometer, Observation and the MeterFilter API — low-cardinality keys versus high-cardinality keys.
• Prometheus, histograms and histogram_quantile — why the quantile is an estimate and why buckets matter.
• Google SRE workbook, the chapters on service-level objectives — SLI, SLO, and error budget, which Lesson 62 picks up as a design exercise.
• Brendan Gregg’s USE method and Tom Wilkie’s RED method — resource signals versus request signals.
• Next in this path: Microservices Security and Zero Trust. The trace you just designed will show a 401; the next lesson decides who is allowed to call whom.
• Module 5 applies the same contract: Design a Payment Processing System, Design a Notification Platform, Design a Large-Scale E-Commerce Platform, and Design a Real-Time Dispatch Platform.