OpenTelemetry

Introduction

OpenTelemetry (OTel) is the vendor-neutral standard for tracing, metrics, and logs — replacing fragmented agents (Jaeger client, Prometheus client, custom MDC) with one SDK and OTLP export. Distributed tracing propagates trace context across Spring Boot services, Kafka consumers, and JDBC calls — one trace ID follows a payment request from gateway to database.

Java integration: OpenTelemetry Java agent (zero-code attach) or Spring Boot 3 Micrometer OTel bridge. Export to Grafana Tempo, Jaeger, Datadog, or Honeycomb via OTLP. Staff engineers mandate trace propagation before microservice count exceeds five — otherwise incidents are blind.

Business problem

Operating microservices without observability:

• Blind incidents: Payment timeout — which of 8 services slow? Hours of log grep.
• No correlation: Separate trace ID, request ID, log MDC — can't link metrics spike to failing traces.
• Vendor lock-in: Datadog agent + Jaeger + Prometheus — three SDKs, three configs.
• Missing SLO data: No histogram of checkout latency — guessing p99 from averages.
• Kafka black box: Consumer lag visible but which message processing fails? — need span per consume.

Why this topic exists

OpenTelemetry unifies the three pillars with one context propagation model:

• Traces: Spans with parent-child hierarchy — visualize request waterfall across services.
• Metrics: Counters, gauges, histograms — RED (Rate, Errors, Duration) per service.
• Logs: Structured logs with trace_id injected — click trace → see logs; click log → see trace.
• Context propagation: W3C traceparent header — gateway → order → payment → Kafka consumer.
• OTLP: Single export protocol to any backend — switch vendor without re-instrumenting app.

Core concepts

OpenTelemetry concepts for Java:

• Tracer: Creates spans — span = tracer.spanBuilder("processPayment").startSpan().
• Span: Named operation with start/end time, attributes (http.status_code=200), events.
• Trace: Tree of spans sharing trace_id — root span from gateway, child spans per service call.
• Context propagation: Inject traceparent into HTTP headers and Kafka record headers on produce; extract on consume.
• Meter: Records metrics — counter.add(1, attributes) for orders_processed_total.
• Auto-instrumentation: Java agent instruments Spring MVC, JDBC, Kafka, HttpClient without code changes.

Internal architecture

Distributed trace across Java microservices:

Client POST /checkout
    │
    ▼ trace_id=abc123 span_id=root
┌─────────────┐
│ API Gateway │  span: HTTP POST /checkout (120ms)
└──────┬──────┘
       │ traceparent: 00-abc123-root-01
       ▼
┌─────────────┐
│ Order Svc   │  span: createOrder (80ms)
│             │    ├─ span: JDBC insert (15ms)
│             │    └─ span: kafka send order.events (5ms)
└──────┬──────┘
       │ Kafka header: traceparent
       ▼
┌─────────────┐
│ Inventory   │  span: consume OrderCreated (40ms)
└─────────────┘

OTLP export → OpenTelemetry Collector → Tempo (traces)
                                      → Prometheus (metrics)
                                      → Loki (logs via trace_id)

Tracing, metrics, logs, and propagation:

Code walkthrough

Manual OpenTelemetry spans in Java + Spring Boot 3 Micrometer:

• try (Scope scope): Makes span current — child HTTP/JDBC auto-instrumentation nests under it.
• setAttribute: Searchable facets in trace UI — filter by payment.currency=USD.
• recordException: Stack trace attached to span — visible in Jaeger/Tempo.
• sampling.probability: 10% in prod reduces cost — always sample errors if tail-based sampling configured.

// build.gradle — Spring Boot 3.2+
dependencies {
    implementation("io.opentelemetry:opentelemetry-api:1.36.0")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("io.micrometer:micrometer-tracing-bridge-otel")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
}

// Manual span in payment logic
@Service
public class PaymentService {
    private final Tracer tracer = GlobalOpenTelemetry.getTracer("payment-service");

    public PaymentResult charge(PaymentRequest req) {
        Span span = tracer.spanBuilder("charge")
            .setSpanKind(SpanKind.SERVER)
            .startSpan();
        try (Scope scope = span.makeCurrent()) {
            span.setAttribute("payment.amount", req.amount());
            span.setAttribute("payment.currency", req.currency());
            validate(req);
            PaymentResult result = gateway.charge(req);
            span.setStatus(StatusCode.OK);
            return result;
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            throw e;
        } finally {
            span.end();
        }
    }
}

// application.yml — OTLP export
management:
  tracing:
    sampling:
      probability: 1.0   # 100% dev; 0.1 prod
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces
  metrics:
    export:
      otlp:
        enabled: true

Production example

Java agent auto-instrumentation + Kafka propagation:

• -javaagent: Zero-code instrumentation for Spring, JDBC, Kafka, RestTemplate — fastest adoption path.
• OTEL_SERVICE_NAME: Appears in every span — filter traces by service in UI.
• Kafka inject/extract: Consumer continues same trace — end-to-end checkout visibility.
• traceId in logs: MDC populated by Micrometer bridge — correlate logs to traces in Grafana.

# Dockerfile — attach OpenTelemetry Java agent
ENV OTEL_SERVICE_NAME=payment-service
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
ENV OTEL_TRACES_EXPORTER=otlp
ENV OTEL_METRICS_EXPORTER=otlp
ENV OTEL_INSTRUMENTATION_KAFKA_ENABLED=true
ENTRYPOINT ["java", "-javaagent:/otel/opentelemetry-javaagent.jar", \
  "-jar", "/app.jar"]

# Kafka producer — propagate trace context in headers (agent does auto)
# Manual if needed:
public ProducerRecord<String, OrderEvent> withTrace(OrderEvent event) {
    ProducerRecord<String, OrderEvent> record =
        new ProducerRecord<>("order.events", event.orderId(), event);
    GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
        .inject(Context.current(), record, (carrier, key, value) ->
            carrier.headers().add(key, value.getBytes(StandardCharsets.UTF_8)));
    return record;
}

# Grafana dashboard — RED metrics from Micrometer
# rate(http_server_requests_seconds_count{service="payment"})
# histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))

# Structured logging with trace_id (Logback)
# logging.pattern.level=%5p [traceId=%X{traceId} spanId=%X{spanId}]

Enterprise case study

Stripe — observability culture: Stripe invests heavily in distributed tracing for payment flows — every API request traceable end-to-end. Java teams adopting OTel mirror this: sampling strategy is critical — 100% traces at Stripe-scale is petabytes; use probabilistic sampling (1–10%) plus tail-based sampling (keep all errors and slow traces). Lesson: define SLO on trace-derived metrics — p99 checkout span duration < 500ms.

• Before: Logs only — MTTR 2 hours for cross-service timeout incidents.
• After: OTel traces — identify slow JDBC in inventory service in 5 minutes.
• Java agent rollout: Attach agent via ENV in Deployment — no code change first sprint.
• Cost control: 10% head sampling + collector tail sampling for status=ERROR.

Performance considerations

OTel performance impact on Java:

• Sampling: Head-based 10% sampling — 90% requests zero export overhead.
• Batch export: OTLP exporter batches spans — async export thread, minimal request path impact.
• Agent overhead: Typically <3% CPU — measure in load test before prod.
• Cardinality explosion: Don't put user_id on metric labels — unbounded series kills Prometheus.
• Span attributes: Limit high-cardinality attributes on spans — use logs for detail.

Security considerations

Observability security:

• PII in spans: Never attribute email, PAN, password — use opaque IDs only.
• OTLP TLS: Encrypt export to collector — mTLS in zero-trust environments.
• Trace data retention: Traces may contain URLs with tokens — scrub query params.
• RBAC on trace UI: Production traces visible to on-call only — not all engineers.

Scalability considerations

Scaling observability pipeline:

• Collector fan-out: App → regional collector → vendor backend — buffer and batch at scale.
• Tail sampling at collector: Keep errors and >1s latency — drop happy path bulk.
• Metrics aggregation: RED per service — don't export raw span as metric per request at scale.
• Log volume: Structured JSON logs with trace_id — ship to Loki/ELK with retention policy.

Production challenges

Real OTel adoption challenges:

• Broken propagation: Missing traceparent on Feign call — orphan spans; fix with agent or manual interceptor.
• Async gaps: @Async without context propagation — child work detached trace; use Context.taskWrapping.
• Dual instrumentation: Old Jaeger + new OTel — duplicate spans; migrate fully, one SDK.
• Sampling bugs: 0% sampling in prod — no traces; verify OTEL_TRACES_SAMPLER config.
• Clock skew: Cross-service span timestamps misordered — use NTP on nodes.

Common mistakes

• No context propagation across Kafka/async — traces fragment into useless single-service spans.
• 100% sampling in production — cost explosion and collector overload.
• High-cardinality metric labels (userId, orderId) — Prometheus OOM.
• PII in span attributes — compliance violation in trace storage.
• Ignoring logs correlation — traces without trace_id in logs — half the debugging value lost.

Debugging guide

Debug OpenTelemetry in Java services:

• Verify export: curl http://otel-collector:4318/v1/traces — collector health; check app logs for OTel export errors.
• Agent debug: OTEL_JAVAAGENT_DEBUG=true — verbose agent startup (dev only).
• Trace not appearing: Check sampling probability, firewall to collector, wrong endpoint port (4317 gRPC vs 4318 HTTP).
• Broken parent: Inspect incoming request for traceparent header — missing at gateway = root only.

# Enable OTel logging (dev)
export OTEL_JAVAAGENT_DEBUG=true
export OTEL_TRACES_EXPORTER=logging  # console spans

# Spring Boot — verify tracing active
curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
  http://localhost:8080/actuator/health

# Collector pipeline check
kubectl logs deployment/otel-collector -n observability

Best practices

• Adopt W3C tracecontext (traceparent) — standard propagation header.
• Start with Java agent auto-instrumentation — add manual spans for business operations only.
• Set OTEL_SERVICE_NAME per Deployment — clear service identity in traces.
• Use probabilistic sampling in prod (1–10%) + tail sampling for errors.
• Inject trace_id into structured logs — Micrometer MDC bridge.
• Propagate context through Kafka headers and @Async executors.
• Define SLOs from trace metrics — p99 latency, error rate per endpoint.

Anti-patterns

• Custom X-Request-Id only — no W3C propagation — incompatible with OTel ecosystem.
• Manual span for every method — noise; focus on boundaries and business ops.
• Exporting traces directly to vendor from every pod — bypass collector buffering.
• Storing full request body in span attributes — size limits and PII risk.
• Different trace IDs in logs and traces — failing to enable Micrometer-OTel MDC integration.

Staff engineer notes

• Staff engineers mandate trace propagation checklist in microservice PR template — HTTP, Kafka, async covered.
• Observability is not optional at 5+ services — OTel before optimizing performance you can't measure.
• Sampling is a product decision — 10% sample with error tail capture balances cost and debuggability.
• Metrics for alerting, traces for debugging, logs for detail — three pillars, one trace_id thread.
• OpenTelemetry Collector is the control point — switch Tempo to Jaeger without touching Java apps.

Interview questions

Hands-on exercise

Lab: OpenTelemetry concepts:

• Run playground — review trace hierarchy simulation.
• Write traceparent header format (version-traceId-spanId-flags).
• List 3 span attributes for payment charge operation.
• Explain head vs tail sampling trade-off for prod.
• Bonus: sketch OTLP flow app → collector → Tempo.

Press Run to execute. Real javac + JVM via remote API. Falls back to simulator on network failure.

Architecture trade-offs

• Java agent vs manual SDK: Agent wins speed; SDK wins control and smaller footprint.
• Head vs tail sampling: Head wins simplicity; tail wins keep-all-errors without 100% cost.
• OTLP to collector vs direct vendor: Collector wins flexibility; direct wins simplicity for small setups.
• 100% traces vs 10% sample: Full wins debuggability; sample wins cost at scale.

Summary

OpenTelemetry is the observability foundation for Java microservices — traces show where latency lives, metrics fire alerts, logs provide detail, all linked by trace_id. Deploy the Java agent, fix propagation across Kafka and async, and sample intelligently before production traffic grows. You now have the platform stack: microservices, Kafka, Redis, Docker, Kubernetes, and OTel.

Key takeaways

• OpenTelemetry — unified traces, metrics, logs with OTLP export.
• Distributed tracing — W3C traceparent propagation across HTTP and Kafka.
• Java agent — fastest Spring Boot instrumentation path.
• Sampling — probabilistic in prod; tail sample errors at collector.
• Correlate logs with trace_id — complete the debugging story.