Service Discovery and Configuration Management

Advanced System Design & Software Architecture · Lesson 39 · Microservices and Enterprise Architecture

Lesson Overview

Lesson 38 extracted payment from the commerce deployable. Checkout still has to find a live payment process, and payment still has to learn its timeouts, its provider URL, and its credential without those values being baked into the image. In the modular monolith those were a method call and a property file on the same classpath. A second process makes both of them operational problems. A stale answer to either one is an incident: traffic sent to a pod that is still starting, or a capture attempted with yesterday's provider host and today's idempotency key.

Discovery is how a caller gets a current address for a service. Configuration is how a process gets the values that are not part of its artifact. They fail differently. Discovery fails when the address is missing, stale, or points at a process that cannot do the work. Configuration fails when a new pod cannot start, when a running pod keeps a value the operator thinks they changed, or when a secret is sitting in a git repository next to a feature flag. Service Discovery and the enterprise pair Service Discovery and Service Registry name client-side and server-side lookup. Twelve-Factor Services states the config rule at the level of a principle. This lesson picks a concrete design for TechLearningPro on Kubernetes, says when a registry would replace DNS, and separates reviewed non-secret config from credentials.

The design is deliberately small. Checkout calls http://payment.commerce.svc.cluster.local. Kubernetes' Service and the readiness probe are the registry. There is no Eureka cluster and no Consul catalog in the default path, because a Service name already does server-side discovery for a cluster that has DNS and endpoints. Spring Cloud Config, reached through spring.config.import, supplies reviewed non-secret values at startup and fails the pod if it cannot. Provider credentials never enter that git-backed store. They arrive as a Kubernetes Secret mounted only into the payment deployment. Live refresh is reserved for values whose mid-process change is safe, such as a log level. Timeout budgets that lesson 41 will treat as part of the release change with the binary, by rollout, so the running code and the budget stay one thing you can diff.

Learning Objectives

By the end of this lesson you will be able to:

1. Distinguish server-side discovery, client-side discovery, and a headless DNS record, and place Kubernetes Service DNS in that set.
2. Explain why a readiness probe that calls the card provider removes healthy capacity during a provider outage.
3. Configure a Spring Boot process to import Spring Cloud Config at startup, fail closed when that import is required, and retry the import with spring-retry.
4. Separate @ConfigurationProperties rebinding from @RefreshScope, and state what a refresh does not do to a client built once in @PostConstruct.
5. Keep provider secrets out of the config repository and out of catalog pods, and lock down the actuator surfaces that would print them.
6. Decide which settings may change without a restart, and which must roll with the deployment.
7. Describe Consul's agent, catalog, and passing-only health query as the registry you would add on virtual machines, including the failure you take on by running it.

Prerequisites

• Lesson 38's payment process and the PaymentCaptures port. This lesson does not reconsider that boundary.
• Lesson 25's load balancing vocabulary: a stable virtual address in front of instances. Load Balancing and Traffic Management is the edge and the cluster balancer. Discovery here is how an in-cluster caller names payment at all.
• A working picture of Kubernetes Services, endpoints, and probes. You do not need to design a service mesh. Service Mesh is a later placement option, not a prerequisite.
• Spring Boot externalized configuration: environment variables override files, @ConfigurationProperties binds a prefix, and management.server.port can differ from server.port.
• Lesson 32's warning that a dependency being down is not the same as a decline. Readiness and circuit breaking answer different questions, and this lesson keeps them apart so lesson 41 has somewhere to live.

Detailed Core Concepts

What discovery is asking

A caller has a logical name, payment, and needs an IP address and a port that will accept a capture request now. Something must maintain the set of acceptable addresses and must remove a process that should not receive work. The design choices are where that set lives and who picks the address.

Server-side discovery. The caller uses a stable name. A load balancer or a packet filter behind that name chooses an instance. The caller does not know how many pods exist. Kubernetes Service of type ClusterIP is this pattern. CoreDNS answers payment.commerce.svc.cluster.local with the Service's virtual IP. kube-proxy, IPVS, or a CNI data plane forwards the packet to an endpoint in the Service's EndpointSlice. The EndpointSlice contains pod IPs whose readiness condition is true. The caller caches the virtual IP and that cache is harmless, because the virtual IP does not change when a pod is replaced. The set of pods changes behind it.

Client-side discovery. The caller asks a registry for instances and picks one. Spring's DiscoveryClient is the abstraction: getServices() and getInstances(String serviceId) return ServiceInstance values with host, port, metadata, and URI. A @LoadBalanced RestClient.Builder from Spring Cloud LoadBalancer uses that list and treats the host in http://payment/captures as a service id. This is justified when there is no local load balancer, or when the client must choose on metadata (a version, a zone) that the balancer does not understand. It costs a client cache. A removed instance can remain in the list until the cache refreshes, so the call needs a connect timeout that fails quickly and a retry policy that does not multiply load. Lesson 41 owns that policy. This lesson only insists the timeout exists.

Headless DNS. A Kubernetes Service with clusterIP: None publishes pod A records directly. The client, or a library, receives several IPs. That is client-side selection with DNS as the registry. It is the right tool for a StatefulSet that must reach a specific pod, or for a client that does its own sharding. It is the wrong default for payment. The JDK caches positive lookups according to networkaddress.cache.ttl. On OpenJDK that default is 30 seconds when no security manager is installed, and a value of -1 caches forever. A cached pod IP from a headless record can outlive the pod. The ClusterIP name does not have this hazard, because the address you cache is the virtual one. Do not switch payment to headless "to skip kube-proxy" unless the client refreshes endpoints on a timer you measured and the connect timeout is short.

A registry process. Consul and, historically, Eureka keep a catalog the client queries. Use one when the workloads are virtual machines with no Service abstraction, or when several clusters must share one catalog. Do not add one beside Kubernetes DNS for a single cluster "because microservices need a registry." You would then have two sources of truth, and they will disagree. Service Registry describes the component. The component is already present as the EndpointSlice controller.

Spring Cloud Netflix Eureka is in maintenance. New work on this platform does not start an Eureka server. @EnableEurekaClient in an old sample is history, not a step in the design.

Readiness is membership, not dependency health

A pod is in the EndpointSlice when its readiness probe succeeds. kubelet calls the probe. Spring Boot exposes probe groups when management.endpoint.health.probes.enabled=true (Boot turns this on when it detects Kubernetes). Point the probe at /actuator/health/readiness on the management port.

Readiness means "this process can accept work." For payment, that includes its own database, because a capture cannot be recorded without it. A database check in the readiness group is reasonable: if PostgreSQL is unreachable, this pod should leave the Service and another replica, or none, should take the call. The saga then sees a connection failure and stores UNCERTAIN or retries within its budget, which is an explicit outcome.

Readiness must not call the card provider. If it does, a provider outage marks every payment pod unready. The Service empties. Checkout has nobody to connect to, so it cannot record an idempotency key and reconcile. You turned a partial dependency failure into a total disappearance of the service that knew how to be uncertain. Provider health belongs in the client policy of lesson 41: a timeout, a breaker that opens on provider errors, and a result of UNCERTAIN. Liveness is a third question: "should kubelet kill this process?" A stuck deadlock qualifies. A slow provider does not. Killing the pod on a provider timeout restarts a healthy JVM and drops in-flight reconciliations.

yamlCopy
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8081
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8081
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8081
  failureThreshold: 30
  periodSeconds: 2

The startup probe keeps liveness from killing a slow JVM before it listens. failureThreshold times periodSeconds must cover boot, including the config-server retry below.

Configuration is not one bucket

Build-time constants belong in the image: the code, the default log configuration structure, the name of the application. They change when you release.

Environment identity belongs in the process environment: the namespace, the config-server URL, spring.profiles.active, the name that selects a Kubernetes Secret. These are known before the process can talk to anything. Kubernetes injects them at pod start. Updating a ConfigMap key that was injected as an environment variable does not change the running process. The kernel gave the process its environment at exec time. A rollout is how that class of change ships.

Reviewed runtime policy belongs in a store with history: timeout budgets you are willing to reload, feature flags that are not secrets, log levels. Spring Cloud Config with a git backend gives you a pull request for the change. The server serves /{application}/{profile}/{label}. The client names itself with spring.application.name. Shared application.yml on the server is visible to every application that imports the server. That shared document is a dangerous default. Payment's provider URL must not live in a file every consumer of the server receives. Put shared values only where sharing is the point (a company tracing endpoint). Put payment's values in payment.yml, selected because the application's name is payment.

Secrets belong in a store with access control and rotation, not in git. A Kubernetes Secret in the payment namespace, sourced from an external secrets mechanism your platform already runs, mounted into the payment pod only. Spring Cloud Config can decrypt {cipher} values when the server holds encrypt.key. That feature is real and it concentrates every decrypted secret on the config server. TechLearningPro does not put the provider credential in the config repository, encrypted or not. A compromise of the config server should reveal timeout numbers, not the credential that captures cards.

Startup import, fail closed, retry

Spring Cloud Config Client binds the server through Boot's config data import:

yamlCopy
spring:
  application:
    name: payment
  config:
    import: "configserver:http://config-server.platform.svc.cluster.local"
  cloud:
    config:
      retry:
        max-attempts: 6
        initial-interval: 1000
        multiplier: 1.2
        max-interval: 4000

Omitting the optional: prefix means the process refuses to start when the server cannot be reached. The documentation's fail-fast behavior with spring.config.import is that omission. spring.cloud.config.fail-fast=true is the older bootstrap switch; with the import syntax, leaving off optional: is the switch that matters. Retry requires spring-retry and AOP on the classpath. The defaults are six attempts, starting at one second, multiplier 1.1. The block above makes the budget explicit.

optional:configserver: is the wrong prefix for payment. A pod that starts on packaged defaults after the config server blipped will capture money with the timeouts from the image, which may be the developer's laptop values. Checkout's price-feature flags can be argued either way; payment's capture path fails closed. Existing pods are unaffected by a config-server outage. They already have their environment. The outage blocks new pods and rollouts. That is why the config server runs at least two replicas, with a disruption budget, and why you do not roll it during enrollment week. The startup probe's budget must cover the retry sleep, or kubelet will kill the container while spring-retry is still backing off.

The config server's address is a Kubernetes DNS name injected as configuration. It is not looked up through Spring Cloud Config. A design that discovers the config server via the same mechanism the config server enables is a cycle. Break it with the environment.

bootstrap.yml and a parent bootstrap context are the legacy path. Boot 2.4 and later, which is what this course standardizes on, uses spring.config.import. A sample that only works because spring-cloud-starter-bootstrap is on the classpath is a sample to retire, not to copy. If both are present, know which one won before you debug a value that "will not change."

Refresh, and what it does not refresh

POST /actuator/refresh on the management port reloads the environment and refreshes @RefreshScope beans. @ConfigurationProperties beans are rebound by ConfigurationPropertiesRebinder on the environment-change event, unless the bean is also @RefreshScope, in which case it follows the scope: the proxy drops its target and the next call creates a new one. Rebinding runs initialization callbacks, including @PostConstruct, on the properties bean.

That sounds more powerful than it is. A RestClient built in another bean's @PostConstruct from a URL it copied into a field will keep the old URL. The properties object changed. The client did not. Either read the URL from the properties bean on each call, or put the client bean in @RefreshScope and accept that in-flight calls see either the old or the new target, not a mixture inside one method if you read the fields once at the start of the method. Dave Syer's notes on this behavior are the right mental model: two calls on a refresh-scoped proxy can hit two targets if a refresh lands between them. For a capture, that is unacceptable if the two targets are two provider environments. So payment's provider URL and its timeout budget do not refresh live. They roll with the deployment. Log level can refresh. A boolean that only affects a log line can refresh.

/actuator/busrefresh from Spring Cloud Bus fans a refresh out through a broker. That adds a control-plane dependency. A rolling restart is easier to reason about for a credential or a money-path timeout. The bus is not part of the payment design.

Actuator exposure

/actuator/env and /actuator/configprops will show the values you worked to segregate, including anything that did end up in the environment. Spring Boot's management.endpoint.env.show-values is never, always, or when-authorized. Set when-authorized, and do not expose env, refresh, or configprops on the public ingress. Bind the management port to a different number and a network policy that admits kubelet and the platform's operators, not the shopper gateway.

yamlCopy
management:
  server:
    port: 8081
  endpoint:
    env:
      show-values: when-authorized
    health:
      probes:
        enabled: true
  endpoints:
    web:
      exposure:
        include: health,info,refresh

refresh on that list is acceptable on the management port. It is not acceptable on the Service port that checkout calls. An unauthenticated refresh is a way for anyone who can reach the pod to change log levels and, worse, to force re-resolution of whatever you unwisely made refreshable.

Consul, when you are not on Kubernetes

Some TechLearningPro workloads still run as virtual machines: a partner-file importer in a network that is not the cluster. There, a Service DNS name does not exist. Consul is a real registry for that world. Server agents store the catalog with Raft. Client agents run beside the workload, and the workload registers with the local agent. Health checks are HTTP, TCP, gRPC, script, TTL, or Docker. DNS is served on port 8600. HTTP GET /v1/health/service/payment?passing=true returns only instances whose checks pass. Registration is PUT /v1/agent/service/register against the local agent. Clients should use blocking queries (?index=&wait=) so they long-poll instead of hammering the catalog.

The failure modes are the ones you took on by running it. The local agent dies, and with it the registration and the checks that agent was performing, even if the process is healthy. A partition between client agents and servers flaps health. A check that calls the card provider has the same bug as the bad readiness probe: the catalog empties during a provider incident. Point the HTTP check at the process's readiness path, which checks the local database and not the provider. Consul's session and lock API is a different product surface. Distributed Locking and Coordination is where locks belong. Do not use a service-catalog health check as a lock, and do not use a Consul lock as a substitute for the reservation row in lesson 32.

Spring Cloud Consul will register a Spring Boot app and expose DiscoveryClient if you choose that stack (spring.cloud.consul.host, discovery health-check path). Prefer the platform DNS name when you have Kubernetes. Prefer Consul when you do not. Prefer neither in the same process "for portability." Two registries means two answers.

Architecture and Design Diagrams

mermaidCopy
flowchart TD
    checkout[Checkout pod]
    dns[CoreDNS]
    svc[Service payment ClusterIP]
    slice[EndpointSlice ready pods only]
    pay1[Payment pod ready]
    pay2[Payment pod not ready]
    cfg[Config server git backend]
    secret[Payment namespace Secret]
    provider[Card provider]
    checkout -->|payment.commerce.svc| dns
    dns --> svc
    svc --> slice
    slice --> pay1
    slice -.->|excluded| pay2
    cfg -->|import at start name equals payment| pay1
    secret -->|mounted only here| pay1
    pay1 --> provider

Checkout never receives a pod IP from a Java registry. The not-ready pod stays out of the slice because its readiness probe failed, for example while it is still retrying the config server. The Secret arrow does not touch checkout or catalog.

mermaidCopy
sequenceDiagram
    participant Kubelet
    participant Pay as Payment pod
    participant Cfg as Config server
    participant Ep as EndpointSlice
    participant Co as Checkout
    Kubelet->>Pay: start container
    Pay->>Cfg: spring.config.import
    alt config server unreachable
        Pay-->>Pay: retry then exit
        Note over Ep: pod never ready
    else config loaded
        Pay-->>Kubelet: readiness 200
        Kubelet->>Ep: add pod IP
        Co->>Pay: POST capture via ClusterIP
    end

A config-server failure prevents membership. It does not eject pods that already loaded their configuration. That asymmetry is the availability property you operate: old replicas keep capturing, new replicas wait.

Technical Explanation

Why the ClusterIP cache is safe and the headless cache is not

Trace a rolling deploy. Pod A is ready. Pod B starts, loads config, passes readiness, and joins the EndpointSlice. Pod A then terminates: it is removed from the slice when it stops being ready, and kube-proxy stops sending it packets. Checkout's resolver still holds the Service's ClusterIP. Packets to that IP land on B. No Java code cleared a cache, because no Java code stored a pod address.

Now suppose someone marks the Service headless so a client load-balancer can "see the pods." Checkout's first DNS answer is pod A's IP. The JDK caches it for networkaddress.cache.ttl. A terminates. For the rest of the TTL, checkout connects to an IP that resets or, worse, has been recycled by the CNI for an unrelated pod. The bug looks like intermittent connection resets at deploy time, and it will be blamed on the application. Keep the ClusterIP. If a future mesh sidecar owns discovery, the application still calls the Service name and the sidecar owns the endpoints. Service Mesh is that placement. Adopting it to solve a problem ClusterIP already solves is a second control plane.

What the config server actually returns

For an application named payment and a profile prod, the server composes documents from the git label (branch or tag): shared defaults, then profile-specific shared files, then payment.yml, then payment-prod.yml. Later documents override earlier keys. The label the client requests must be explicit in deployment configuration (spring.cloud.config.label), not "whatever the server's default-label currently is." A server default that moved from master to main has stranded clients before. Pin the label in the Deployment. Review the git repository like code. A force-push that rewrites a label rewrites production configuration.

search-paths on spring.cloud.config.server.git can restrict which directories the server reads. Use it so a sandbox team's files are not on the server's search path for production. The native profile (spring.profiles.active=native on the server, with spring.cloud.config.server.native.search-locations) reads a local directory. That is a test double. A production config server whose "git" is a single disk on one replica will lose the review trail and will not be highly available.

Binding values without copying them

javaCopy
@ConfigurationProperties(prefix = "payment.capture")
public record CaptureSettings(Duration connectTimeout, Duration responseTimeout, URI providerBaseUri) {}

yamlCopy
payment:
  capture:
    connect-timeout: 200ms
    response-timeout: 800ms
    provider-base-uri: https://provider.example/v2

The record is immutable, which is what you want: a refresh replaces the bound object rather than mutating fields under readers. The HTTP adapter reads settings.connectTimeout() when it builds a request, or the RestClient is created in a @Bean method that receives CaptureSettings and the whole bean is recreated only on rollout. Do not cache providerBaseUri in a static field "for speed." The call rate does not justify a stale static.

The client timeouts are the ones that release a connection. Lesson 41 will repeat this from the resilience side. A config value of response-timeout: 30s copied from a sample will hold checkout's threads or virtual threads, and the HTTP connection pool, across a provider stall. The config review is part of the latency budget. A pull request that raises response-timeout above the saga's payment budget is a defect, not a tuning exercise.

Secrets in the process, not in the document

The provider credential is a file mount or an environment variable from a Secret. @ConfigurationProperties can bind PAYMENT_PROVIDER_KEY from the environment. The config server's payment.yml does not contain a key, a {cipher} blob, or a comment with a sample key that happens to be real. Catalog's Deployment does not name the Secret. A network policy prevents catalog's namespace from reading payment's secrets. The test is a manifest check in CI: the catalog Deployment's volumes and env sources do not include payment-provider.

Rotation is a rollout or a dual-key window the provider supports. Hot refresh of a signing key under an in-flight request produces signature failures that look like a provider outage and will open the circuit breaker for the wrong reason. Schedule the rotation. Do not put the key behind /actuator/refresh.

Discovery of payment from checkout, in code

javaCopy
@Bean
RestClient paymentRestClient(RestClient.Builder builder) {
    var factory = new SimpleClientHttpRequestFactory();
    factory.setConnectTimeout(Duration.ofMillis(200));
    factory.setReadTimeout(Duration.ofMillis(800));
    return builder
        .baseUrl("http://payment.commerce.svc.cluster.local")
        .requestFactory(factory)
        .build();
}

SimpleClientHttpRequestFactory.setConnectTimeout(Duration) and setReadTimeout(Duration) are Spring Framework 6.1 APIs. The base URL is the Service name. There is no DiscoveryClient injection. A @LoadBalanced builder would be appropriate if you had deliberately chosen client-side selection. Adding it "in case we leave Kubernetes" hides the real dependency, which is the cluster DNS name, and introduces a cache. YAGNI applies to registries.

spring.application.name of checkout is checkout. Its config import is the same pattern, with its own document checkout.yml. It does not import payment's document. If a key you need is missing, fail startup for the money path rather than defaulting a timeout to zero, which in some HTTP clients means "wait forever." Validate the record with @Validated and @NotNull so a missing duration fails the bind.

JVM DNS and deploy spikes

Even with ClusterIP, a pathological networkaddress.cache.ttl=-1 is wrong the day the Service is deleted and recreated and its virtual IP changes. Set the TTL to a small positive number in the container image (30 seconds matches the OpenJDK default, which is fine) and do not set -1 because a blog warned about DNS storms. A Service lookup per request is cheap beside a card capture. An infinite cache is not cheap during a Service recreation. Headless records, if you ever use them for a StatefulSet, need a short TTL and a client that can tolerate a stale IP for that long.

Practical Implementation or Design Walkthrough

Problem

Payment is a new deployment. Checkout must send captures to a ready pod only. Payment must refuse to start without its reviewed timeouts and must obtain the provider credential without that credential appearing in git or on catalog pods. A config-server blip during a scale-up must not produce pods that capture with default timeouts.

Requirements

• New payment pods do not enter the Service until config has loaded and the local database check passes.
• A provider outage does not empty the Service.
• Checkout uses the Service DNS name and a connect timeout of 200 milliseconds, response timeout of 800 milliseconds, both shorter than the saga step budget.
• payment.yml in the config git repo contains no secret. The catalog manifest does not reference the payment Secret.
• /actuator/env is not on the public port, and show-values is when-authorized.
• Config-server unavailability fails new payment pods after a bounded retry. Pods that have already started keep their loaded values.

Steps

1. Deploy the config server as two replicas behind config-server.platform.svc, git backend, label pinned by clients. Confirm a bad git label fails the client at startup rather than silently using a default branch.
2. Add payment.yml with payment.capture.connect-timeout, response-timeout, and provider-base-uri. Review it like code. Do not add the key.
3. Set spring.config.import without optional:, plus retry, plus spring-retry on the classpath. Set the startup probe so the retry can finish.
4. Mount the provider Secret only on the payment Deployment. Bind it into CaptureSettings or a small ProviderCredential properties type that is excluded from toString() and from logs. Never log the properties object wholesale.
5. Point readiness at /actuator/health/readiness including the database health indicator, excluding any custom indicator that calls the provider. Point checkout's RestClient at the Service name with the factory timeouts above.
6. Network-policy the management port. Expose health, info, and refresh there only.

Failure handling

Config server down at scale-up: new pods exit after retry, the HorizontalPodAutoscaler keeps the old pods, and an alert on "desired replicas greater than ready" fires. You do not flip the import to optional: to make the alert stop. Provider down: pods stay ready, checkout's client times out at 800 milliseconds, the adapter returns UNCERTAIN, the saga reconciles. A pod that cannot open its own database leaves the Service. If every payment pod loses its database, the Service is empty and checkout fails fast on connect. That empty Service is correct. There is no payment process able to record a capture. Pretending otherwise would accept charges you cannot store.

Validation

• Scale payment to a new pod with the config server scaled to zero. The pod is not in the EndpointSlice. Restore the config server. The pod becomes ready without an image change.
• Point a temporary readiness indicator at the provider sandbox and block the sandbox. Confirm the EndpointSlice drops the pod, then delete that indicator. This is a test you run once to prove the failure mode, not a probe you leave in place.
• GET the public Service port's /actuator/env and expect a refusal. GET it on the management port without credentials and expect values to be withheld.
• Grep manifests: provider key material is absent from the config repo and from non-payment Deployments.
• During a rolling restart, a capture client that resolves the Service once per process still succeeds. A test build that sets networkaddress.cache.ttl to a negative value and uses a headless Service should fail a chaos case you keep as documentation, not as the production Service type.
• Change a log level through refresh and confirm the provider base URI did not change without a rollout. The second half is the assertion that matters.

Enterprise Real-World Scenarios

Scenario 1: Enrollment morning, config server rolled with the application

Context. Traffic matches lesson 23's peak. An operator rolls the config server to pick up a one-line timeout change and, in the same window, the autoscaler adds checkout and payment pods.

What breaks. New pods block on spring.config.import. The startup probe's budget was copied from a sample (thirty seconds) and the retry sleeps longer than that, so kubelet kills the container, which starts again and hammers the config server. Ready replicas stay at the old count. If someone "fixes" it by setting optional:configserver:, the new payment pods start with image defaults, including a thirty-second read timeout from a tutorial, and checkout's threads pile up on the provider. The saga's uncertain branch, which assumed an 800 millisecond ceiling, now waits thirty seconds per attempt.

Design. Config server changes are a different change window from enrollment, with two replicas and a pod disruption budget so one replica stays up. Timeout changes for payment go through a payment rollout after the event, not through a hot refresh. The startup probe is calculated from the retry budget, not copied. optional: is forbidden on payment by a config lint in CI.

Scenario 2: One shared application.yml on the config server

Context. The first version of the config repo had a single document because there was a single monolith. After lesson 38, payment and catalog both import the server. Someone adds provider-key to the shared file so payment can boot, encrypted with {cipher} and encrypt.key on the server.

What breaks. Every application that imports the server receives the property. Catalog logs its environment on a crash, and the crash log contains the decrypted value if a binder put it on the environment and the log pattern prints it. The config server's key is now a crown jewel: whoever can read the server's memory can read every ciphered property. Rotation means re-encrypting the git history, which still holds the old ciphertext, while the old key must remain to read it.

Design. payment.yml holds non-secrets only. The credential is a Secret in the payment namespace, injected by the platform's external-secrets path, absent from catalog. encrypt.key is not the payment strategy. Audit the server's git history for the word key before the first production capture. Twelve-Factor Services said config comes from the environment. It did not say all config shares one document, and it did not say secrets and flags are the same environment entry with the same readers.

Best Practices

• Use the Kubernetes Service DNS name for in-cluster calls. Add a Java DiscoveryClient when the platform does not already provide server-side discovery.
• Keep ClusterIP for payment. Treat headless Services as a deliberate client-side choice with a short DNS TTL.
• Readiness includes local dependencies the process cannot work without, such as its own database. It excludes downstream services whose failure the caller must handle.
• Liveness restarts a wedged process. It does not restart a process whose dependency is slow.
• Import config without optional: on money paths. Budget the startup probe for the retry.
• Pin spring.cloud.config.label. Do not depend on the server's default branch.
• Put secrets in a namespaced Secret, not in the config git repo and not in {cipher} blobs on a shared server.
• Roll provider URLs, credentials, and timeout budgets. Refresh log levels.
• Read configuration from the bound object at use time, or recreate the client on rollout. Do not copy values into static fields.
• Expose env and refresh on the management port only, with show-values: when-authorized.
• One registry per environment. DNS plus Consul plus Eureka is three answers.
• Alert on desired replicas above ready replicas during a config-server incident. Do not paper over it with optional import.
• Keep the config server's availability independent of the services it configures, and do not discover the config server through itself.

Common Mistakes and Anti-Patterns

Eureka beside Kubernetes. Two catalogs, two caches, and a client that load-balances to an instance the EndpointSlice has already dropped. The Service name was sufficient. A registry you do not operate will still page you when it is down, and payment will refuse to start for a reason unrelated to payment.

Readiness that calls the provider. The Service empties exactly when you need a process alive to record UNCERTAIN and to reconcile. The probe made a downstream failure look like "no instances." Checkout then cannot tell "nobody is running" from "the provider is slow."

optional:configserver: on payment. The pod starts. The timeouts are whatever the image shipped. The incident appears later as hung captures, not as a failed deploy. Fail the deploy.

Environment variable from a ConfigMap, then "we updated the ConfigMap." The running JVM keeps the old environment. Either mount a file and re-read it under a defined refresh policy, or roll the pod. Operators who think the edit is live will debug the wrong binary.

Refresh of a copied URL. @ConfigurationProperties rebound, the RestClient still holds the old base URL, and half the fleet has moved. Captures straddle two provider environments. Roll the client with the value.

Secrets in the shared config document. Every importer becomes a reader. Crash logs and /actuator/env become exfiltration paths. Separate the Secret, restrict the actuator, and keep catalog's manifest clear of payment's credential.

Infinite JVM DNS cache with headless records. Deploys black-hole traffic for the life of the process. This is a configuration bug, not a reason to abandon DNS. Fix the TTL or stop using headless records for a stateless payment tier.

Health check as a lock. Consul or a readiness gate used to ensure "only one capturer runs" will drop capacity and will not enforce the idempotency key. The key and the unique constraint in payment's database are the lock. Lesson 36's coordination tools are for a different problem.

Troubleshooting and Design Review

Design review: What is the stable name the caller uses, and who updates the set of addresses behind it? What would empty that set, and is that condition "cannot do local work" or "a downstream is unhappy"? Where does a new pod get non-secret config, and does startup fail if that source is down? Where does the credential live, and which deployments can mount it? Which values are allowed to change without a restart, and which client objects would keep the old value? Is there a second registry that can disagree with the first?

Interview Questions and Answers

Q1. Where should TechLearningPro's checkout find the payment service on Kubernetes, and why not a Spring DiscoveryClient?

Checkout calls the ClusterIP Service name payment.commerce.svc.cluster.local. CoreDNS returns a stable virtual IP. The EndpointSlice and the data plane select a ready pod. That is server-side discovery, and the platform already runs it. DiscoveryClient.getInstances("payment") plus a @LoadBalanced client is a second selector with its own cache. It is the right tool on virtual machines that have Consul or a similar catalog and no Service. On this cluster it duplicates the EndpointSlice and can send traffic to a pod the slice has already removed. Service Discovery presents both models. The choice here is the one that adds no new process.

Q2. Why is caching the ClusterIP safe when caching a headless Service's A records is not?

The ClusterIP does not change when pods are added or removed. kube-proxy or the CNI updates the forwarding rules. A cached virtual IP still reaches a current ready pod. A headless Service returns pod IPs. The JDK caches them for networkaddress.cache.ttl (OpenJDK's default is 30 seconds without a security manager; -1 means forever). After a rollout those IPs can be dead or reused. Payment stays on ClusterIP so deploy-time client caches are not part of the failure mode.

Q3. What belongs in a readiness probe for payment, and what must stay out?

The process can accept work: the HTTP server is up and its own database is reachable. A provider call stays out. If readiness depends on the provider, a provider outage removes every pod from the Service, and checkout can no longer record an uncertain capture against a live payment process. Liveness is only for a wedged JVM. A startup probe covers a slow boot, including config-import retries, so liveness does not kill the container during backoff. Downstream failure is a client timeout and a saga status, which is lesson 41 and lesson 32, not a membership event.

Q4. How does a Spring Boot service load Spring Cloud Config without bootstrap.yml?

spring.config.import=configserver:http://config-server.platform.svc.cluster.local, with spring.application.name selecting the document. Leaving off the optional: prefix fails startup when the server is unreachable, which is the import-style equivalent of fail-fast. Add spring-retry and set spring.cloud.config.retry.max-attempts, initial-interval, multiplier, and max-interval so the failure is bounded and visible. Pin spring.cloud.config.label. Do not discover the config server via the config server. The legacy bootstrap context is optional extra machinery; this course does not require it.

Q5. What is the difference between ConfigurationPropertiesRebinder and @RefreshScope?

On an environment refresh, the rebinder re-initializes @ConfigurationProperties beans so holders of those beans see new values. A @RefreshScope bean is a proxy whose target is destroyed and created again on the next call. A properties bean that is also refresh-scoped follows the scope, not the rebinder. Neither path rewrites a RestClient or a URL that some other bean copied into a field at @PostConstruct. For provider URL and capture timeouts, roll the deployment. Use refresh for log levels and similar values that cannot redirect a capture.

Q6. Why are secrets excluded from the config git repository even with {cipher}?

{cipher} and encrypt.key decrypt on the config server, so the server becomes the place where every secret is recoverable. The shared document is delivered to every application that imports the server, so catalog would receive payment's credential. Git history keeps old ciphertext. TechLearningPro puts non-secrets in payment.yml and the provider key in a Secret mounted only by payment. Actuator env stays on the management port with management.endpoint.env.show-values=when-authorized, so a crash or a curious GET is not a second distribution channel.

Q7. A colleague updates a ConfigMap that is injected as environment variables and expects running pods to change. What actually happens?

Nothing inside the process. The environment block is fixed at exec. File mounts of a ConfigMap can update on the kubelet's sync, and even then the JVM will not notice unless it re-reads the file or you refresh a bean that reads the file. For capture settings, roll the pods. Document the path so the next incident is not a debate about whether the edit "should have" been live.

Q8. When would you introduce Consul?

When the workload cannot use a Kubernetes Service: virtual machines in a network without that control plane, needing a catalog, DNS on port 8600, and GET /v1/health/service/{name}?passing=true. Register with the local agent, health-check the process's own readiness, and use blocking queries. You then operate Raft servers, agent failure, and flap. You do not also run Consul for services that already have EndpointSlices. You do not use Consul sessions as the seat lock; that remains the reservation row.

Q9. Config server is down during scale-out. Old payment pods are fine, new ones are not. Is that a bug in fail-closed import?

No. Pods that finished import keep the values they loaded. Pods that cannot import must not capture with image defaults, so they stay unready. The operational duty is to run the config server as two replicas, avoid rolling it at the peak, and alert when ready replicas fall behind desired. Switching to optional: would start pods that violate the timeout budget. The asymmetry between old and new pods is the property you want, and it is why a config-server outage is a deploy incident rather than a silent behavior change.

Q10. How does this lesson constrain the resilience policies in the next lessons?

Discovery membership is "can accept local work." It is not the circuit breaker. The RestClient connect and read timeouts configured here are the numbers lesson 41's retry budget has to fit inside, and they must be shorter than the saga step. A retry that ignores them will multiply calls to a pod the EndpointSlice still correctly includes. Configuration review includes those durations. A thirty-second sample timeout is a design defect even if the Service discovery is perfect. The public contract of payment, if any partner calls it, is lesson 40; inside the cluster, checkout's port from lesson 38 stays the contract, and the Service name is not a public URL.

Hands-On Exercises

Exercise 1: Membership and import

Problem. Payment must join the Service only after it has loaded payment.yml, and must stay in the Service when the card provider is black-holed.

Tasks.

1. Write the spring.config.import line and the retry settings. State what optional: would change.
2. Specify readiness, liveness, and startup probes, including which one includes the database and how the startup budget relates to retry.
3. Draw the sequence from container start to the first checkout call, and the sequence when the provider times out.
4. List the manifest checks that keep the provider Secret off catalog.

Deliverables. The YAML fragments, the two sequences, and the check list.

Exercise 2: A registry you do not need, and one you do

Problem. One engineer wants Eureka in the cluster. Another has a file importer on virtual machines that must call payment.

Tasks.

1. Reject or accept Eureka for in-cluster checkout, with the cache and the dual-catalog failure.
2. Sketch the Consul registration for the importer's network: which check URL, which query, what the local agent's death does to the catalog entry.
3. Say how the importer learns the config-server URL without using the config server to find the config server.

Rubric.

Lesson Summary

Checkout finds payment through the Kubernetes Service name. Ready pods are the membership set, and readiness means the process and its own database can work, not that the card provider is healthy. Headless DNS and a Java DiscoveryClient add a cache of pod addresses this design does not need. Consul is the alternative where no Service exists, with checks aimed at the process, not the provider.

Payment loads reviewed, non-secret configuration with spring.config.import and refuses to start if the config server is unreachable. Credentials live in a Secret mounted only into payment. Refresh is for log levels. Provider URLs and timeout budgets roll with the deployment, because a RestClient built from a copied value will not follow a rebound properties bean. The management port is where env and refresh live, with values shown only when authorized.

Those timeouts are the budget Circuit Breaker, Retry and Bulkhead Patterns must respect. The public shape of an order API, including how a breaking change is versioned, is API Design, Versioning and Contracts. Discovery made the process reachable. It did not define what the bytes mean.

Further Reading

• Spring Cloud Config client reference: spring.config.import, fail-fast behavior, and spring.cloud.config.retry.*.
• Spring Cloud Commons, ConfigurationPropertiesRebinder and @RefreshScope.
• Kubernetes documentation: Service, EndpointSlice, readiness, liveness, and startup probes; DNS for Services and Pods.
• HashiCorp Consul: agent service registration and GET /v1/health/service/:name?passing.
• Spring Framework SimpleClientHttpRequestFactory connect and read timeouts; org.springframework.cloud.client.discovery.DiscoveryClient.
• OpenJDK networking properties: networkaddress.cache.ttl.
• This course: Microservices Architecture Principles, Load Balancing and Traffic Management, Distributed Transactions and Saga Pattern.
• Primers: Service Discovery, Twelve-Factor Services, Service Registry, Service Mesh.