Service Registry

Introduction

A service registry is the authoritative datastore of service instances, metadata, and health status that discovery clients query — Eureka, Consul, etcd (K8s), and ZooKeeper (legacy) fill this role. Stripe's internal service catalog registers every production microservice with owner, SLO tier, and dependency graph for incident routing and discovery.

Real production story

Stripe's 2017 incident: a payments worker could not reach the Ledger service — engineers grep'd Slack for "ledger hostname" and found three conflicting answers. No central registry meant new services launched without registering, load balancers pointed at decommissioned ASGs, and the service dependency graph existed only in a stale wiki page.

Platform team shipped an internal Service Registry backed by Consul: services register on boot with metadata (team, tier, on-call, OpenAPI URL), health checks every 10s, deregister on graceful shutdown. Discovery clients and the incident bot query the same source. Unknown caller attempting Ledger connection without registry entry triggers alert — shadow deployments caught before production traffic.

Business problem

Business pressure: Stripe runs 1000+ internal services — without registry, discovery drift causes payment path failures and incident confusion ("who owns this endpoint?").

• Revenue at risk: Unregistered Ledger endpoint → payment capture retries exhaust → merchant-visible failures.
• Engineering velocity: New service bootstrap must include registry registration — paved road template.
• Compliance / trust: SOC2 requires inventory of production services — registry is source of truth.

Architecture overview

Registry stores: service name, instance ID, host/port, health status, metadata (version, zone, team). Registration: self-registration (Eureka heartbeat) vs third-party (K8s controller watches pods).

• Definition: Authoritative directory of service instances and metadata.
• When to adopt: Client-side discovery, multi-cluster, need rich metadata beyond DNS.
• When to defer: Pure K8s with DNS-only — Endpoints object is implicit registry.
• Operability: Registry cluster HA, backup, and watch API rate limits monitored.

Architecture motivation

Why architects care: Registry is the system of record for "what exists, who owns it, is it healthy" — discovery clients are readers; registration is writer contract every service must honor.

• Force: Ephemeral instances need authoritative membership list.
• Constraint: Registry must be HA — outage cannot stall all east-west traffic.
• Outcome: Single catalog for discovery, incident routing, dependency analysis, and compliance.

Internal architecture

Stripe Consul service registry topology:

  ┌─────────────┐     register/heartbeat     ┌─────────────────┐
  │ Payment svc │ ────────────────────────────────▶│ Consul cluster  │
  │ instance A  │     meta: team=payments,tier=0   │  (3-node HA)    │
  └─────────────┘                                └────────┬────────┘
  ┌─────────────┐     register/heartbeat                  │
  │ Ledger svc  │ ────────────────────────────────▶         │
  │ instance 1  │                                         │
  └─────────────┘                                         │
                                                          │ watch/query
  ┌─────────────┐     discover ledger.prod                │
  │ Payment svc │ ◀───────────────────────────────────────┘
  │  (client)   │     returns healthy instances only
  └─────────────┘

  Incident bot queries: ledger.prod → owner: ledger-team → PagerDuty
  Compliance export: all tier-0 services with last heartbeat

Data flow

Registration path: Ledger service boots → registers ledger.prod with IP:port, metadata, HTTP health check URL → Consul marks passing → Payment client watch fires → adds endpoint to load balancer pool.

• Register: Self-register on boot with TTL heartbeat; miss 3 heartbeats → auto-deregister.
• Discover: Client long-poll watch on service name; cache locally; reconnect on disconnect.
• Deregister: SIGTERM handler deregisters before shutdown; health check fail removes within 10s.

System design diagram

Two diagrams show the Service Registry topology and the primary request/event path used in production at scale.

Production code example

Consul self-registration with health check — Stripe service bootstrap pattern:

// service bootstrap — register on start, deregister on shutdown
import consul from "consul";
import { createServer } from "http";

const agent = consul({ host: process.env.CONSUL_AGENT, promisify: true });
const SERVICE_ID = `${process.env.SERVICE_NAME}-${process.env.HOSTNAME}`;

async function register() {
  await agent.agent.service.register({
    id: SERVICE_ID,
    name: process.env.SERVICE_NAME, // e.g. ledger.prod
    address: process.env.POD_IP,
    port: parseInt(process.env.PORT, 10),
    tags: ["tier:0", `team:${process.env.TEAM}`, `version:${process.env.GIT_SHA}`],
    check: {
      http: `http://${process.env.POD_IP}:${process.env.PORT}/health`,
      interval: "10s",
      timeout: "3s",
      deregistercriticalserviceafter: "30s",
    },
  });
}

async function deregister() {
  await agent.agent.service.deregister(SERVICE_ID);
}

process.on("SIGTERM", async () => {
  await deregister();
  server.close();
});

const server = createServer(app);
server.listen(process.env.PORT, async () => {
  await register();
  console.log(`Registered ${SERVICE_ID} with Consul`);
});

Enterprise case study

Stripe internal service catalog (2017–2019): Consul registry reduced "unknown endpoint" incident MTTR from 45 min to 8 min and automated SOC2 service inventory.

• Before: Hostnames in wikis; three conflicting Ledger URLs; no dependency graph.
• Decision: Mandatory Consul registration in service template; metadata schema (team, tier, on-call).
• After: Incident bot auto-pages owner; compliance export nightly; zero unregistered tier-0 services.

Trade-offs

• Self-registration vs sidecar: Self-register simpler; sidecar (Consul agent) handles health without app code.
• CP vs AP registry: Consul CP for consistency; Eureka AP for partition tolerance — match to discovery needs.
• Registry scope creep: Storing config in registry — prefer dedicated config service; registry for location + metadata.

Security considerations

Security is architectural: Registry ACLs control who can register as production service name — prevent namespace hijacking.

• Identity: mTLS between agents; only authorized nodes register as ledger.prod.
• Data: Metadata may contain internal URLs — registry API not public internet.
• Audit: Registration events logged — who registered unknown instance when.

Scalability analysis

Scale dimensions: Stripe shards Consul by environment; 10k+ instances per cluster with gossip optimization.

• Horizontal scale: Multi-node Consul cluster; clients connect to local agent.
• Hot spots: Popular service watch fanout — agent cache reduces server QPS.
• Cost: Registry HA cluster baseline — justify with incident time saved and compliance automation.

Failure scenarios

What breaks: Registry partition, ghost instances, registration forgotten on new service launch.

• Ghost instances: Crash without deregister — TTL heartbeat evicts after timeout; tune aggressively.
• Registry outage: Clients use cached endpoints — stale but available; alert on cache age.
• Split metadata: Wiki says owner A, registry says owner B — registry wins; wiki deprecated.

Staff engineer insights

• Service registry vs service discovery: registry is the phone book; discovery is looking up a number — interviewers test this distinction.
• Stripe paved road: CI fails if new service lacks registry registration in deploy manifest.
• Rich metadata (owner, tier, dependencies) turns registry from ops tool into architecture governance.

Interview questions

Architecture review questions

• Does every production service register on boot and deregister on graceful shutdown?
• Are health checks evicting failed instances within defined SLA (e.g. 30s)?
• Is registry cluster deployed HA with monitored watch/query latency?
• Does registry metadata include owner, tier, and on-call for incident routing?
• Do clients cache registry data with fallback behavior during registry outage?
• Are registry ACLs preventing unauthorized service name registration?

Summary

Service registry at Stripe scale is the HA system of record for instance location, health, and ownership metadata — powering discovery clients, incident bots, and compliance exports. Staff architects mandate registration in service templates, tune heartbeat eviction, and extend registry beyond IP/port into architecture governance.