Observability

Introduction

Observability is the ability to infer internal system state from external outputs — metrics, logs, traces, and events — to answer novel questions during incidents. LinkedIn's feed and messaging platforms generate petabytes of telemetry; observable architecture designs instrumentation, cardinality, and SLO dashboards as first-class deliverables.

Real production story

LinkedIn messaging saw elevated delivery latency for three days before a Sev-2 was declared — averages looked fine while p99 doubled for EU users. Root cause was a new Kafka consumer group lagging silently; alerts targeted mean latency on US dashboards only. The observability overhaul mandated RED metrics per service, trace propagation across gRPC hops, SLO burn alerts on p99 by region, and "observability readiness" as a launch gate. MTTR for similar incidents dropped from days to under an hour.

Business problem

LinkedIn operates real-time professional graph products globally. Without observability built into architecture, teams ship blind — incidents become multi-day hunts and postmortems repeat the same "missing dashboard" theme.

• Member experience: Silent latency regressions erode feed engagement before error rates spike.
• Engineer toil: Manual log grep across 200 services does not scale — observability reduces MTTR and burnout.
• Compliance: Audit trails for messaging and ads require structured, retained telemetry.

Architecture overview

Observability goes beyond monitoring known failures — it supports ad-hoc questions via high-cardinality context when incidents surprise you. Staff practice: logs for narrative, metrics for SLOs, traces for latency, events for business audit.

• Three pillars: Metrics (aggregates), logs (detail), traces (causality) — unified by trace_id.
• Golden signals: Latency, traffic, errors, saturation — RED/USE methods per tier.
• SLO-driven alerts: Burn rate alerts on SLI — not static thresholds on CPU.
• Cardinality discipline: High-cardinality labels belong in traces/logs, not metric labels.

Architecture motivation

Observability is a quality attribute: You cannot operate what you cannot see. Architecture must define golden signals, trace context propagation, and alert policies before prod — not after the first Sev-1.

• Force: Microservices multiply failure modes; cross-service debugging requires correlated traces.
• Constraint: Telemetry volume and cost explode at LinkedIn scale — design sampling and cardinality limits.
• Outcome: Any on-call engineer answers "what broke?" within minutes using standard dashboards.

Internal architecture

LinkedIn observability stack integration — architecture embeds telemetry at every hop:

• Trace context must cross async boundaries — inject trace_id in Kafka headers.
• Runbook URL in alert annotation — observability without action is trivia.

Service (auto-instrumented SDK)
   ↓
OpenTelemetry collector (sample · enrich · scrub PII)
   ↓
├─ Metrics → TSDB (Prometheus / M3)
├─ Traces  → Tempo / proprietary
└─ Logs    → Kafka → searchable store
   ↓
SLO recording rules + burn rate alerts
   ↓
Incident workflow (PagerDuty · runbook links in alert)

Data flow

Telemetry flow: Request generates span → metrics incremented at boundaries → structured log with trace_id → collector aggregates → SLO dashboard updates → burn alert if budget consumed.

• Ingress: Edge generates trace root; propagate W3C traceparent to all hops.
• Async: Domain events carry trace_id and causation_id for pipeline debugging.
• Storage: Metrics long-retention; traces sampled (1–10%) with tail-based sampling for errors.

System design diagram

Two diagrams show the Observability topology and the primary request/event path used in production at scale.

Production code example

OpenTelemetry gRPC middleware — production Java instrumentation pattern:

• Standard middleware ensures every service emits consistent RED metrics without copy-paste.
• Pair with platform-enforced trace propagation headers on gRPC and Kafka.

public final class ObservabilityServerInterceptor implements ServerInterceptor {
  private final Meter meter;
  private final Tracer tracer;

  @Override
  public <ReqT, RespT> Listener<ReqT> interceptCall(
      ServerCall<ReqT, RespT> call,
      Metadata headers,
      ServerCallHandler<ReqT, RespT> next
  ) {
    String method = call.getMethodDescriptor().getFullMethodName();
    Span span = tracer.spanBuilder(method)
        .setParent(Context.current())
        .startSpan();
    LongCounter errors = meter.counterBuilder("grpc.errors").build();
    LongHistogram latency = meter.histogramBuilder("grpc.latency_ms").build();

    long start = System.nanoTime();
    Context ctx = span.storeInContext(Context.current());

    return Contexts.interceptCall(ctx, call, headers, next).withOnComplete(() -> {
      latency.record((System.nanoTime() - start) / 1_000_000);
      span.end();
    }).withOnCancel(() -> {
      errors.add(1, Attributes.of(AttributeKey.stringKey("method"), method));
      span.recordException(new StatusRuntimeException(Status.CANCELLED));
      span.end();
    });
  }
}

Enterprise case study

LinkedIn messaging latency observability program: Regional p99 blind spot delayed incident detection.

• Before: US-centric dashboards; EU p99 2× US for 72h unnoticed; MTTR 3 days.
• Decision: Regional SLO slices, trace propagation standard, observability launch checklist.
• After: EU regression detected in 12 minutes; MTTR under 1h; telemetry cost flat via sampling.

Trade-offs

• Cardinality vs debuggability: Per-user metric labels break TSDB — use traces for drill-down.
• Sampling vs cost: 100% trace retention is prohibitive — tail-sample errors and high-latency spans.
• Auto-instrumentation vs custom spans: Auto covers frameworks; business spans need manual hooks for domain KPIs.
• Centralized vs federated dashboards: Golden dashboards per service; avoid one unmaintainable mega-board.

Security considerations

Telemetry is sensitive: Logs and traces capture PII and secrets — observability architecture includes scrubbing and access control.

• PII scrubbing: Collector redacts email, token, message body before storage.
• RBAC: Trace access limited by team; break-glass audited for cross-team incidents.
• Secret leakage: CI tests reject log statements with credential patterns.

Scalability analysis

Observability at LinkedIn scale is a cost and performance concern — unbounded telemetry can dwarf application infra spend.

• Ingest limits: Per-service quotas; drop debug logs in prod automatically.
• Aggregation: Pre-aggregate business metrics at edge; avoid shipping raw click streams to alerts.
• Retention tiers: Hot 7d full fidelity; warm 90d aggregated; cold archive for compliance.

Failure scenarios

Observability failures cause operational failures: Alert fatigue, missing regional slices, and broken trace propagation extend incidents.

• Alert storm: 500 pages per incident — consolidate to SLO burn and symptom-based alerts.
• Broken traces: Missing propagation across Kafka — async hops appear as orphan spans.
• PII leak in logs: Scrubber failure in collector — block deploy if PII regex tests fail.

Staff engineer insights

• If your architecture diagram has no observability box on every arrow, the design is incomplete.
• Mean latency is a lie at scale — architect alerts on p99/p999 and SLO burn by region and tenant tier.
• Observability readiness should block launch like security review — shipping without RED metrics is technical debt with interest due at 2am.

Interview questions

Architecture review questions

• Are RED/USE metrics exported from every service endpoint?
• Is W3C trace context propagated across sync and async hops?
• Are SLOs defined with multi-window burn rate alerts?
• Is PII scrubbing verified in CI for log and trace payloads?
• Do alerts link to runbooks and ownership in service catalog?
• Is observability readiness part of launch checklist for tier-0?

Summary

Observability at LinkedIn scale means architecture that makes unknown failures debuggable: unified traces, RED metrics, SLO burn alerts, PII-safe pipelines, and launch gates — because operations without telemetry is guesswork at petabyte scale.