Service Discovery

Introduction

Service discovery lets microservices find each other's current network location (IP, port, health) without hardcoded hostnames — essential when containers scale and reschedule constantly. Google's internal Stubby/gRPC resolution and Kubernetes DNS provide dynamic lookup for thousands of internal services.

Real production story

Google's early Borg deployments hardcoded service hostnames in flag files — when a Search indexing job rescheduled to new machines, downstream callers pointed at dead IPs until flags propagated (15–45 min drift). A Maps routing service outage traced to stale endpoint lists in a config file edited by hand.

Google built Stubby (gRPC predecessor) with integrated name resolution via Borg naming (BNS) — callers ask for logical name /prod/maps/routing, resolver returns current healthy backends, load balances, and watches for membership changes. Kubernetes adopted similar patterns: ClusterIP DNS resolves to current pod endpoints; Istio adds virtual service routing. Hardcoded IPs became a lint violation.

Business problem

Business pressure: Google services autoscale and reschedule by the second — static service directories cannot keep pace with Borg/K8s churn.

• Revenue at risk: Stale endpoints in Ads bidding path reduce auction participation — direct ad revenue impact.
• Engineering velocity: Manual hostname updates block every deploy — discovery must be automatic.
• Compliance / trust: mTLS cert SANs must match discovered identities — discovery and identity are linked.

Architecture overview

Discovery patterns: client-side (client queries registry, caches, load balances — Eureka, gRPC resolver), server-side (load balancer fronts pool — ALB + K8s Service), DNS-based (CoreDNS, Route53). Hybrid common at scale.

• Definition: Registry of service name → healthy instance endpoints, updated on register/deregister/health change.
• When to adopt: Dynamic schedulers (K8s, Borg, ECS); more than handful of services.
• When to defer: Fixed VM fleet with stable IPs — DNS A records sufficient.
• Operability: Discovery lag metrics — time from pod ready to resolver cache update.

Architecture motivation

Why architects care: Service discovery decouples logical service identity from physical location — prerequisite for autoscaling, rolling deploys, and multi-region failover.

• Force: Ephemeral infrastructure — IPs are meaningless across minutes.
• Constraint: Sub-millisecond lookup on hot paths — cache and watch, not query per request.
• Outcome: Clients resolve logical names; platform handles membership changes.

Internal architecture

Google / Kubernetes service discovery stack:

  Caller (Ads bidding service)
         │
         │ gRPC target: dns:///payments.prod.svc.cluster.local
         ▼
  ┌──────────────────┐
  │  CoreDNS / BNS   │  watches K8s Endpoints / Borg cell
  └────────┬─────────┘
           │ returns A records / SRV
           ▼
  ┌──────────────────┐
  │ Client-side LB   │  gRPC pick_first / round_robin
  │ + health watch   │  xDS (Istio) push endpoint updates
  └────────┬─────────┘
           │
     ┌─────┴─────┐
     ▼           ▼
  Pod 10.0.1.5  Pod 10.0.2.8
  (payment-v3)  (payment-v3)

  On deploy: old pods deregister → watch fires → clients drop stale IPs
  No caller redeploy required

Data flow

Primary path: Bidding service opens gRPC channel to payments:50051 → resolver queries K8s Endpoints API (cached) → returns 3 pod IPs → client LB picks one → mTLS handshake with SPIFFE ID → RPC.

• Registration: K8s kubelet registers pod on Ready; deregisters on SIGTERM + grace period.
• Health: Only Ready pods in Endpoints; failing readiness removes from discovery immediately.
• Watch path: Long-poll/watch Endpoints; client cache updates in <2s on membership change.

System design diagram

Two diagrams show the Service Discovery topology and the primary request/event path used in production at scale.

Production code example

Kubernetes Endpoints watch + gRPC resolver — Google/gRPC ecosystem pattern:

// gRPC Go — custom resolver using K8s Endpoints watch
type k8sResolver struct {
  cc     resolver.ClientConn
  client kubernetes.Interface
  svc    string
  ns     string
}

func (r *k8sResolver) start(ctx context.Context) {
  watcher, _ := r.client.CoreV1().Endpoints(r.ns).Watch(ctx, metav1.ListOptions{
    FieldSelector: fields.OneTermEqualSelector("metadata.name", r.svc).String(),
  })
  for event := range watcher.ResultChan() {
    ep := event.Object.(*corev1.Endpoints)
    var addrs []resolver.Address
    for _, subset := range ep.Subsets {
      for _, addr := range subset.Addresses {
        for _, port := range subset.Ports {
          addrs = append(addrs, resolver.Address{
            Addr: fmt.Sprintf("%s:%d", addr.IP, port.Port),
          })
        }
      }
    }
    r.cc.UpdateState(resolver.State{Addresses: addrs})
  }
}

// Channel uses logical target — no hardcoded IPs
conn, _ := grpc.Dial("dns:///payments.prod.svc:50051",
  grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`))

Enterprise case study

Google Borg → gRPC name resolution (2008–2015): BNS eliminated manual flag files; became foundation for gRPC resolver API used industry-wide.

• Before: Hand-edited hostname flags; 15–45 min stale endpoint drift; Maps outage.
• Decision: Logical naming (BNS), client-side resolution with watch, integrated health.
• After: Zero caller redeploy on callee scale events; gRPC resolver pattern open-sourced.

Trade-offs

• Client-side vs server-side LB: Client-side (gRPC) smarter routing but SDK complexity; server-side (ALB) simpler clients, extra hop.
• DNS TTL vs watch: DNS caching causes stale records — prefer watch-based (xDS, Endpoints watch) for gRPC.
• Registry SPOF: Eureka/consul cluster must be HA — discovery outage stalls all east-west traffic.

Security considerations

Security is architectural: Discovery must integrate with identity — only register authenticated workloads with valid SPIFFE/K8s SA.

• Identity: mTLS cert SAN matches discovered service identity — prevent rogue registration.
• Data: Registry ACLs — only namespace X can register as payments.prod.
• Supply chain: Compromised pod registering as payment service — admission controller validates SA + image signature.

Scalability analysis

Scale dimensions: Google BNS handles millions of names; K8s clusters shard DNS by namespace and use NodeLocal DNS cache.

• Horizontal scale: More pods auto-register; discovery watch pushes delta, not full scan.
• Hot spots: Thundering herd on cold cache — prefetch endpoints on channel create.
• Cost: Control plane QPS to API server from Endpoints watches — tune watch efficiency at 5k+ pod clusters.

Failure scenarios

What breaks: Stale cache after deploy, split registry views, discovery during network partition.

• Stale DNS: Client caches dead IP for TTL — connection errors until refresh; use low TTL or watch.
• Split brain registry: Multi-region Eureka without sync — route to wrong region; prefer K8s-native or global xDS.
• Graceful drain miss: Pod killed before deregister — in-flight requests fail; preStop hook + readiness fail first.

Staff engineer insights

• Hardcoded service URLs in config files are a discovery anti-pattern — lint them in CI like secrets.
• Google staff loop: explain difference between service discovery and service registry — discovery is the client act, registry is the data store.
• Readiness probe timing is part of discovery — NotReady pods must never appear in endpoint lists.

Interview questions

Architecture review questions

• Are service locations resolved dynamically — no hardcoded IPs in config?
• Do clients watch registry/endpoints for membership changes (not TTL-only DNS)?
• Are only Ready/healthy instances included in discovery records?
• Is graceful shutdown (preStop + readiness fail) implemented before pod termination?
• Is mTLS identity tied to discovered service name?
• Are discovery lag and stale-endpoint error rates monitored?

Summary

Service discovery at Google scale enables gRPC and microservices to call logical names while Borg/Kubernetes reschedules workloads constantly. Staff architects eliminate hardcoded endpoints, integrate discovery with mTLS identity, and treat readiness probe timing as part of the discovery contract.