Microservices Deployment and Release Strategies

Advanced System Design & Software Architecture · Lesson 46 · Microservices and Enterprise Architecture

Lesson Overview

Deploying the modular monolith was one artifact and one schema migration, and the window was however long Flyway held its lock. Independent services, which is the point of Microservices Architecture Principles, mean several artifacts move on different days, while old and new pods answer the same gateway at the same time. A checkout that starts on version 1.8 and finishes on 1.9 is normal during a rolling update. The release is safe only if 1.8 and 1.9 can both run, and only if the database migration both of them see is compatible with both.

Deploy and release are different acts. A deploy puts bits on pods. A release lets learners exercise a behavior. Kubernetes’ rolling update is a deploy technique: replace pods of a Deployment without a gap, using readiness so traffic does not hit a process that cannot serve. Blue/green and canary are traffic policies on top of a deploy. A feature flag is a release technique that does not require the traffic policy to be all-or-nothing. You can deploy dark code, turn it on for staff, then for five percent of learners, and turn it off without a rollback of the image. You still need the image rollback when the new process crashes on startup. The flag does not replace kubectl rollout undo. The undo does not replace a backward-compatible schema.

This lesson uses the Kubernetes Deployment fields that actually control a rolling update (strategy.rollingUpdate, probes, minReadySeconds, terminationGracePeriodSeconds), Spring Boot’s server.shutdown=graceful, and the expand/contract sequence for schema and API changes. Where a platform already runs Argo Rollouts, the canary steps and the blue/green activeService / previewService fields are the traffic policy. Feature flags go through OpenFeature’s Client.getBooleanValue, not a homemade annotation framework. The SLIs from Observability: Logging, Metrics and Tracing are how a canary earns the next step. The contracts from Microservices Testing Strategies are what must already be true before any of these rollouts start. Module 5’s payment and e-commerce designs will pick a strategy per change. A payment capture path does not get the same canary casualness as a new syllabus layout.

Learning Objectives

By the end of this lesson, you should be able to:

1. Separate deploy from release, and say which of rolling update, blue/green, canary, and feature flag moves bytes versus which moves traffic or behavior.
2. Configure a Kubernetes rolling update so unavailable pods are not sent traffic: readiness, maxUnavailable: 0 when you have the capacity, and a startup probe that does not kill a slow JVM.
3. Drain in-flight requests with preStop, terminationGracePeriodSeconds, and Spring Boot graceful shutdown, in that order.
4. Explain blue/green as two stacks and a cut of the Service or the route, including the database problem a second stack does not solve.
5. Explain a canary as a small percentage of traffic on the new version, promoted by an SLI, and aborted back to the stable weight.
6. Use a feature flag as a release control with an owner and an expiry, and name the four flag kinds you should not mix up: release, experiment, ops, permission.
7. Plan an expand/contract schema and API change so old and new pods overlap safely, and state why a rollback of the application is not a rollback of a destructive migration.
8. Choose a strategy for a Module 5 change — a payment rule versus a storefront experiment — without treating “zero downtime” as a synonym for “no plan.”

Prerequisites

• Lesson 26, Load Balancing and Traffic Management: a Service sends traffic to ready endpoints. This lesson is what “ready” has to mean during a replace.
• Lesson 40, API Design, Versioning and Contracts: expand/contract for JSON fields is the API half of the migration sequence.
• Lesson 43, Observability: Logging, Metrics and Tracing: canary promotion reads the checkout SLI split by a low-cardinality version.
• Lesson 45, Microservices Testing Strategies: can-i-deploy is the contract gate. This lesson is the runtime gate after that.
• Familiarity with a Kubernetes Deployment and a Service. You do not need to have operated Argo.

Detailed Core Concepts

Two versions will run at once, or you do not have a rolling update

A Deployment with strategy.type: RollingUpdate starts new pods before, or as, it stops old ones, bounded by maxSurge and maxUnavailable. maxSurge: 1 and maxUnavailable: 0 means “bring up a new pod first, and never go below the desired count of ready pods.” That requires cluster capacity for one extra pod. maxUnavailable: 1 on a three-replica service is a smaller spike and a real dip in capacity. Pick the one your SLI can absorb. minReadySeconds waits after a pod becomes Ready before the rollout continues, which catches a process that passes the probe and then deadlocks. progressDeadlineSeconds fails the rollout if it cannot make progress, instead of leaving you half-migrated in silence.

Readiness is the load-balancer contract. Kubernetes removes an endpoint when the readiness probe fails. The probe for a Spring Boot service on Kubernetes is /actuator/health/readiness, which Spring Boot exposes when probe endpoints are enabled (it turns them on when it detects Kubernetes). Liveness is /actuator/health/liveness, and it answers “should this process be killed?” A failing database that will recover is a readiness failure, not a liveness failure. Killing the pod because PostgreSQL blipped multiplies the outage. A startup probe covers a JVM that takes longer to open than the liveness threshold would tolerate. Without it, the kubelet restarts a healthy slow start and you watch CrashLoopBackOff on a release that would have worked.

The probe paths are not decorations. A readiness probe that always returns 200, or that hits a path which does not check the things you cannot serve without, will send traffic to a pod that has not run Flyway’s pending migrations or has not warmed the JWKS cache. A readiness probe that checks a downstream you can degrade without will take you out of rotation because payment is down, and then every order pod disappears while payment is the actual incident. Readiness means “this pod can do its job.” It does not mean “the whole company is healthy.”

Drain is a sequence, not a hook you hope runs

When a pod is removed, Kubernetes sends SIGTERM and waits terminationGracePeriodSeconds, then SIGKILL. Spring Boot with server.shutdown=graceful and spring.lifecycle.timeout-per-shutdown-phase stops accepting and waits for in-flight requests up to that timeout. The platform has to stop sending new requests before the grace period starts counting down against live work. The usual pattern is a preStop hook that sleeps a few seconds so the endpoints controller can remove the pod from the Service while it is still serving the requests it already accepted. Then SIGTERM starts graceful shutdown. Then the grace period must be longer than the shutdown timeout, which must be longer than the slowest request you are willing to finish. A checkout capped at a few seconds fits in a 30-second grace. A pod that ignores SIGTERM and gets SIGKILL drops those checkouts and can leave a Kafka producer mid-batch. That is a correctness bug, not a cosmetic one, because of the at-least-once behavior you already designed for.

replicas and a PodDisruptionBudget interact with voluntary disruptions (node drain), not with a bad image. A PDB of minAvailable: 2 on a three-replica Deployment stops a node upgrade from evicting the last spare. It does not stop you from rolling a broken image. The rollout’s readiness rules do that, if you configured them.

Blue/green is a cut, and the database is still shared

Blue/green runs two complete stacks. One is live (blue). The other (green) has the new version and receives no learner traffic until you switch the Service selector, the Ingress backend, or the mesh route. Argo Rollouts expresses this as activeService and previewService: preview is where you run the checks, active is what learners hit, and autoPromotionEnabled: false means a human or a job promotes. Rollback is a switch back, which is fast if green’s bugs appear immediately and if blue’s pods are still up.

The switch is not free of state. Both stacks usually share one database, because two databases of orders would fork the business. A migration that green applied and blue cannot read makes the “instant rollback” a lie. Blue/green buys you a fast traffic cut and a place to run smoke tests against preview. It does not buy you a second schema. It also doubles the idle capacity while both stacks are up, which is a real cost for a large JVM service and a modest cost for a small one. Use it when the cut must be atomic (all learners move together, because mixed versions would corrupt a session or a protocol you failed to make compatible) or when you need a preview environment that is production-shaped. Do not use it as a default for every config tweak. A rolling update is simpler when versions are compatible, which is the constraint you should be meeting anyway.

Session affinity and in-memory sessions break both patterns. A learner whose session lives in pod memory will miss it when the pod is replaced or when the color switches. Sessions belong in a store both versions understand, or the gateway is stateless and the token is the session. Sticky sessions used to “make the rollout work” hide the incompatibility until the sticky pod dies.

Canary is a fraction, and the fraction needs an SLI

A canary sends a small share of traffic to the new version and watches. Argo Rollouts’ canary steps are explicit: setWeight: 5, pause for a duration, setWeight: 25, and an analysis template that queries Prometheus. Promotion is not “we deployed and nobody complained in Slack.” The analysis should use the checkout availability and latency SLIs from Lesson 43, split by a version label that has only two values during the rollout, stable and canary. If the canary’s error rate exceeds the stable rate by a margin you set before you start, the rollout aborts and the weight returns to the stable ReplicaSet.

Five percent of traffic is five percent of checkouts, not five percent of pods, unless every pod carries the same share. Pod-count tricks (“one of twenty pods is new”) approximate a percentage only when the Service balances evenly and the new pod is actually Ready. A canary that is never selected because of a selector typo is a canary that proved nothing. Verify the canary received requests by the version metric before you promote.

Canaries miss bugs that need the other ninety-five percent: rare SKUs, a single tenant, a batch job that does not go through the canary ingress. A canary is a production test of the common path. It is not a proof that a migration job is correct. Data migrations need their own rehearsal, on a copy, with counts.

Do not canary a change that cannot run beside the old version. A new topic name that only the canary produces, while stable consumers still read the old topic, drops the canary’s events on the floor. Both versions have to write something both consumers understand, or the consumer has to be updated first (expand) and the producer second. That is the same expand/contract order as a column.

Feature flags separate release from deploy

A flag is a decision in the process, evaluated per request or per message, with a default when the flag service is down. OpenFeature is the API that keeps that decision out of one vendor’s SDK types. OpenFeatureAPI.getInstance().getClient().getBooleanValue("checkout.tax-v2", false, context) returns the boolean. The targeting key on the EvaluationContext is how you stick a learner to a variant. The default false is the behavior you get when evaluation fails. Choose the default as the safe behavior, which is usually the old path.

Martin Fowler’s four kinds are worth keeping distinct because they have different lifetimes. A release flag hides unfinished behavior and should die when the release is done. An experiment flag assigns cohorts and must be analyzed, not left on. An ops flag is a kill switch for a dependency you may need to shed, and it may live. A permission flag is a product entitlement (this university may use proctoring) and is not a deploy tool. Deleting a permission flag because “flags should be short-lived” removes a product feature. Leaving a release flag for two years creates a matrix of paths nobody tests. The practice is an owner and an expiry on release flags, checked in the same way you check a dependency you meant to upgrade.

Flags have failure modes. Evaluating a remote flag synchronously on the checkout hot path makes the flag service a dependency of every purchase. Cache evaluations, time them out, and fall back to the default. A flag that is on for a canary pod and off for a stable pod is a second source of truth next to the traffic weight. Prefer one mechanism for “who sees the new behavior.” Either the canary pods run the new code unconditionally because the canary is the release, or all pods run both paths and the flag chooses, and the traffic split is only there to limit blast radius of a crash. Doing both without a sentence that explains why means a learner can hit a new pod and still get the old path, and your SLI split by version no longer matches the behavior split.

The flag does not make a schema compatible. Code behind a flag that writes a new NOT NULL column still breaks old pods that do not know the column, the moment any pod runs the migration. Expand/contract first. Then the flag chooses which code path runs on top of a schema both understand.

Expand/contract is the sequence that makes rollback real

Also called parallel change. For a rename of amount to amount_cents on orders:

1. Expand. Add amount_cents as nullable. Deploy code that writes both columns and reads the old one if the new one is null. Old pods still write amount. New pods write both. No pod reads only the new column yet.
2. Backfill. A job copies amount into amount_cents for old rows. It is idempotent. It is throttled. It has a count you can check.
3. Switch reads. New code reads amount_cents and still writes both. You can roll the application back to a version that reads amount, because that column is still populated.
4. Contract. When no deployed version reads or writes amount, a later migration drops it. This deploy cannot be rolled back past the drop. You do not combine step 4 with step 1.

The API version of the same sequence: add a response field, ship clients that tolerate it (consumer pacts with type matchers, not full-body equality), switch clients to require it only after every server writes it, remove the old field in a later contract. A rolling update in the middle of step 1 is safe. A rolling update that starts after someone dropped the column is an outage, and kubectl rollout undo brings back pods that still need the column.

Flyway versioned migrations are forward. V18__add_amount_cents.sql runs once. Editing V18 after it has been applied is ignored or, worse, a checksum failure. The contract step is V22__drop_amount.sql, a new file, after the application versions that needed amount are gone from every namespace, including the canary and the preview stack you forgot. Destructive migrations wait for that inventory.

Rollback is a decision you rehearse

Application rollback: kubectl rollout undo deployment/order-service returns to the previous ReplicaSet if the image is still pullable and revisionHistoryLimit has not discarded it. The previous image must still match the current schema. That is the expand/contract invariant.

Flag rollback: turn the flag off. This is faster than an image rollout when the process is healthy and the bug is in the new path. It does nothing if the new path already wrote data the old path cannot read.

Schema rollback: restore from a backup or run a forward migration that repairs. DROP COLUMN is not undone by an application undo. Practice the restore or do not drop.

A failed rolling update should stop, not loop. If the new pod fails readiness, the Deployment does not shift all traffic. If you set maxUnavailable high and a bug fails slowly, you can still drain capacity. Watch kubectl rollout status. Abort by undoing, not by pushing a third image on top of a half-rolled second one unless the third image is the known-good build.

Architecture and Design Diagrams

Rolling update with a shared schema that both versions understand. The Service has endpoints only for Ready pods. Old and new overlap on purpose.

mermaidCopy
flowchart LR
  GW[Gateway Service]
  GW --> Old1[order 1.8 Ready]
  GW --> Old2[order 1.8 Ready]
  GW --> New1[order 1.9 Ready]
  New1 --> DB[(orders table\namount and amount_cents)]
  Old1 --> DB
  Old2 --> DB

Expand/contract over time. The forbidden move is the diagonal that drops a column while an old pod still serves.

mermaidCopy
flowchart TD
  E[Expand: add nullable column\nwrite both] --> B[Backfill old rows]
  B --> R[Switch reads to new column\nstill write both]
  R --> C[Contract: drop old column\nonly when 1.8 is gone]
  C --> X[Rollback of app no longer enough]

Canary promotion uses the SLI, then either shifts weight or aborts. Blue/green is the same picture with weights at 0 and 100 and a preview probe before the cut.

mermaidCopy
sequenceDiagram
  participant R as Rollout
  participant S as Stable pods
  participant C as Canary pods
  participant P as Prometheus

  R->>C: setWeight 5
  C->>P: checkout SLI version=canary
  S->>P: checkout SLI version=stable
  alt error rate within margin
    R->>C: setWeight 100
  else burn or latency breach
    R->>C: setWeight 0
    Note over S: stable keeps traffic
  end

Technical Explanation

The Deployment fields that decide whether the rollout is safe

yamlCopy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  revisionHistoryLimit: 5
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      terminationGracePeriodSeconds: 40
      containers:
        - name: order
          image: registry.techlearningpro.example/order-service:1.9.0
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            periodSeconds: 10
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]

revisionHistoryLimit: 5 keeps ReplicaSets you can undo to. A limit of 0 leaves you with nothing to undo. The image tag is an immutable version, not latest. latest makes the undo non-deterministic because the tag moves. Spring’s graceful shutdown has to be on in the same release that shortens the grace period, not the release after.

yamlCopy
server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s
management:
  endpoint:
    health:
      probes:
        enabled: true

The 40-second grace period is longer than the 5-second preStop plus the 30-second shutdown wait. If those numbers are inverted, Kubernetes sends SIGKILL while Tomcat still has requests. The checkout SLI will show them as errors. The rollout will look “successful” because the new pods became Ready.

Argo Rollouts, when the Service weight is the product

A canary Rollout replaces the Deployment’s strategy. The stable Service and the canary Service are how the mesh or the ingress splits weight. The analysis template is a Prometheus query you would recognize from Lesson 43, not a synthetic 200 against /health that the broken code still serves.

yamlCopy
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: order-service
spec:
  replicas: 3
  strategy:
    canary:
      stableService: order-stable
      canaryService: order-canary
      steps:
        - setWeight: 5
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: checkout-sli
        - setWeight: 50
        - pause: {duration: 10m}
        - setWeight: 100

Blue/green on the same resource uses activeService and previewService and leaves autoPromotionEnabled false until the preview journey from Lesson 45 has run against preview. Promoting because the pods are Ready skips the only test blue/green added over a rolling update.

If you do not run Argo, a second Deployment and a manual Service selector change is the same blue/green, with more room for a selector typo. Write the cut as a reviewed manifest. Clicking labels in a console at midnight is how preview becomes active by accident.

A flag evaluated with a safe default

javaCopy
public final class TaxPath {
    private final Client flags;
    private final TaxCalculator current;
    private final TaxCalculator revised;

    public TaxPath(Client flags, TaxCalculator current, TaxCalculator revised) {
        this.flags = flags;
        this.current = current;
        this.revised = revised;
    }

    public Money taxFor(OrderDraft draft) {
        EvaluationContext context = new ImmutableContext(draft.learnerId());
        boolean revisedPath = this.flags.getBooleanValue("checkout.tax-v2", false, context);
        return revisedPath ? this.revised.tax(draft) : this.current.tax(draft);
    }
}

getBooleanValue is the OpenFeature Java client call: flag key, default, context. The default false keeps the current calculator if the provider times out. ImmutableContext carries the targeting key so one learner does not flip variants between the quote and the capture. The flag key is a constant you can grep. The revised calculator is the one you delete when the flag’s expiry arrives and the old calculator goes with it. Until then, both are unit-tested. A flag with one untested branch is a branch you will meet during the incident that made you reach for the flag.

The migration files, in order, as separate deploys

V18 adds a nullable column. The application version that writes both columns ships after V18 has run, or in a release where the migrator runs first and the old code tolerates an extra unused column. Old code that uses SELECT * and a positional mapper will break when a column is added. Map by name. That bug is the expand step failing before any contract.

V22 drops the old column only after a query against the cluster shows no pod on an image older than the switch-reads release. Include Jobs and CronJobs in that query. A nightly job on 1.8 will write amount after you dropped it, or will crash and leave settlements unfinished.

Practical Implementation or Design Walkthrough

Problem

Order service 1.9 must store money as integer cents, stop reading the decimal amount, and expose amountCents in JSON while old clients still read amount. Checkout must keep serving during the rolling update. You must be able to undo the application if the new tax path mis-prices, without restoring a database backup.

Constraints

• Three replicas, cluster can schedule one extra pod.
• Shared PostgreSQL. One writer population, two versions during the rollout.
• Checkout SLI: availability and p99, split by version.
• No latest tags. Images 1.8.4 (known good) and 1.9.0.
• A release flag checkout.tax-v2 defaults off. Schema compatibility does not depend on the flag.
• Pact verification from Lesson 45 has already passed for an additive response: both amount and amountCents present.

Sequence

1. Ship V18 adding amount_cents nullable. No application change. Confirm old pods still pass readiness. This is the expand of the schema only.
2. Ship application 1.8.5 if you need a bridge that writes both columns before 1.9, or go to 1.9 if 1.9 is the first writer of both and 1.8 only reads amount and ignores unknown columns. 1.8 must not fail on an unknown column. Verify that on 1.8 against a migrated database in Testcontainers before production.
3. Backfill amount_cents from amount. Publish the row counts: rows where amount_cents is null should hit zero. Keep the job idempotent so a retry does not double.
4. Rolling-update to 1.9.0 with maxUnavailable: 0, the probes and the preStop sleep above, flag default off. 1.9 reads amount_cents when present, otherwise amount. It writes both. Mixed pods are valid.
5. Turn checkout.tax-v2 on for staff, then five percent, watching the SLI. Abort the flag, not the schema, if the tax path is wrong and the process is healthy.
6. If 1.9 crashes readiness, kubectl rollout undo deployment/order-service. 1.8 still reads amount, which is still written. Do not run V22.
7. After 1.8 is gone from Deployments, Jobs, and preview stacks, and clients have had a release that prefers amountCents while still tolerating amount, ship the contract migration that drops amount and remove the old JSON field in a later, separate release. Undo from here is a forward fix or a restore, and you say so in the change record.

What you measure

During step 4, kube_deployment_status_replicas_unavailable stays at zero if the maxUnavailable setting held. The SLI version label shows both versions. A canary or a flag ramp that does not move the version metric is not receiving traffic. During step 5, staff orders are visible in logs by the flag evaluation, not by a second metric per learner. If p99 on the flagged cohort exceeds the threshold, the flag returns to false and you keep 1.9, because the crash is not in process start. If readiness fails, you undo the Deployment and you leave the flag off.

Expected result

Learners buy courses throughout. For a while, rows contain both money representations and responses contain both fields. Rollback of 1.9 returns to 1.8 without a backup. The drop of the old column happens in a later change with an explicit note that application undo is no longer sufficient. The tax experiment ends with the flag deleted, not with a permanent branch in TaxPath.

Enterprise Real-World Scenarios

A payment rule you do not canary casually

Design a Payment Processing System can roll out a new service image with the same rolling-update mechanics, because a stateless adapter should be backward compatible. It should not canary a new rounding rule on five percent of captures while the ledger books the other ninety-five with the old rule, unless finance has accepted a mixed population and a reconciliation report exists for the canary window. Some changes are expand/contract plus a flag that flips for all traffic at a reconciliation boundary (end of day), not a traffic percentage. The case study should say which changes are pod-compatible and which require a cutover window. “Zero downtime” is the wrong requirement for a rule that must be atomic across the ledger. A short, announced freeze with blue/green on the adapter and a single flag flip is more honest than a canary that creates two rounding regimes in one settlement file.

An e-commerce storefront experiment

Design a Large-Scale E-Commerce Platform will want canaries for recommendation ranking and release flags for a new checkout layout. Those are presentation and ranking. The order write path underneath still follows expand/contract, because a layout flag that writes a different order schema is a migration hiding in a product experiment. The case study splits them: flags and canaries on the read and UI path, compatible rolling updates on the order service, and no destructive migration in the same change as the experiment. The SLI for the layout is conversion and error rate, which Lesson 43 would call a product indicator next to the technical availability SLI. Promote the canary only if both hold. A layout that throws no 500s and silently drops payment method upi is a product failure the HTTP availability SLI might miss if you only count 5xx. Include the business result tag in the analysis query.

Best Practices

• Pin image digests or immutable tags. Keep revisionHistoryLimit high enough to undo. Never deploy latest.
• Use readiness for “send traffic,” liveness for “kill the process,” and a startup probe so a slow JVM is not killed for being slow.
• Set maxUnavailable: 0 when you can afford one extra pod and the SLI cannot afford a capacity hole.
• Drain with preStop, then graceful shutdown, with terminationGracePeriodSeconds longer than both.
• Make every overlapping version schema-compatible. Expand, backfill, switch reads, contract later.
• Do not drop a column or an API field in the same release that first adds its replacement.
• Canary against a pre-written SLI, including the business result, and abort on that query.
• Give every release flag an owner and an expiry. Default to the old behavior when evaluation fails.
• Keep permission flags and ops kill switches out of the “delete all flags” cleanup.
• Rehearse undo on a non-production cluster: broken image, flag off, and a migration you intentionally do not roll back.

Common Mistakes and Anti-Patterns

Liveness checks the database. A database blip restarts every pod, which restarts every pool, which deepens the blip. Readiness fails the pod out of rotation. Liveness stays up unless the process is wedged.

maxUnavailable equal to the replica count. The rollout can take every pod down. It is a recreate deploy wearing a rolling name. Set a floor.

Blue/green with a destructive migration in green’s boot. The cut back to blue fails because the column is gone. Run only expandable migrations before the cut. Contract in a later release after blue is decommissioned.

Canary by pod count without checking the metric. The new pod is unready or unselected, you promote anyway, and the “canary” was zero traffic. Require a minimum request count on the canary SLI before the analysis can pass.

A flag and a schema change in one switch. Turning the flag on runs code that assumes V22 has dropped a column the old pods still query. Decouple the flag from the migration. The flag selects a path. The migration is safe for both paths until contract.

Rollback as DROP reversal by hand in production. Ad hoc SQL at midnight is how the backfill gets reversed partially. Forward-fix or restore from a backup you have restored before.

Release flags with no expiry. Two years later, four combinations of tax and layout exist, and the Testcontainers suite covers one. The incident is always in a combination nobody has booted since the quarter it shipped.

Troubleshooting and Design Review

Symptom: rollout sits at one new pod and never finishes. Cause: the new pod is not Ready, or minReadySeconds plus a flapping probe never stabilizes, or progressDeadlineSeconds has not fired yet. Check: pod events and the readiness URL from a debug container. Fix: the probe or the bug. Do not raise maxUnavailable to force the issue out of sight.

Symptom: brief 502s at the start of each pod replacement. Cause: the Service still routes to the pod after SIGTERM, or preStop is missing and the process dies before endpoints update. Fix: the sleep-then-graceful sequence, and a grace period that covers it. Confirm with a load test during a rollout, not only at rest.

Symptom: undo brings back 1.8 and every request 500s on a missing column. Cause: contract migration already ran. Fix: forward migration that restores the column if you still have the data, or a restore. The fix after the fact is worse than the inventory check before V22.

Symptom: canary analysis never promotes and never fails. Cause: the Prometheus query labels do not match the metric, or the pause is waiting for a manual promote you did not document. Fix: run the query by hand before the rollout. A query that returns no series should fail the analysis, not pass it.

Symptom: flag off and the new behavior remains. Cause: the code path does not read the flag, or a second flag service in another region still has the old value, or the canary image has the behavior unconditional. Fix: one evaluation site, a test that the default is the old path, and a log field with the evaluated value on the request you are debugging. Do not log the entire evaluation context if it contains learner data you would not put on a span.

Design review. What is deployed but not released? What runs at the same time, and which schema do both versions tolerate? Where is the SLI query that aborts the canary? What is the undo, and which migration makes undo a restore? Which Jobs are still on the old image? What is the flag’s default when the provider times out, and when will the flag be deleted?

Interview Questions and Answers

Q1. What is the difference between a deploy and a release?

A deploy puts a new image or configuration on the platform. A release lets users exercise a behavior. A rolling update can deploy code that is dark behind a flag, so you have deployed and not released. A flag flip can release behavior without a new image. Blue/green and canary are traffic policies that often do both at once. The distinction matters because rollback of a release may be a flag, while rollback of a crash is an image undo, and neither one undoes a destructive migration.

Q2. What does maxUnavailable: 0 with maxSurge: 1 guarantee, and what does it cost?

Kubernetes starts a new pod before it terminates an old one, and it does not deliberately go below the desired ready count. You need capacity for one extra pod. It does not guarantee the new pod will become Ready, and it does not hide a bad readiness probe. If the cluster cannot schedule the surge pod, the rollout stalls, which is safer than dropping capacity, and it is still an incident if you needed the deploy during an outage.

Q3. Why are liveness and readiness different probes?

Readiness removes the pod from Service endpoints when it should not receive traffic, including during startup and when a required local dependency is missing. Liveness restarts the container when the process is deadlocked or wedged. Pointing liveness at a remote database restarts the fleet when the database blips. Pointing readiness at a global “is the company up?” check removes the fleet during a dependency incident and removes your ability to fail gracefully. Startup probes exist so a slow boot is not treated as a wedge.

Q4. Why do you need preStop if Spring Boot already shuts down gracefully?

Graceful shutdown only helps after SIGTERM. Kubernetes may still route requests to the pod until the endpoints controller observes the terminating state. A short preStop delay keeps the process serving while those endpoints drain, and then SIGTERM triggers server.shutdown=graceful. terminationGracePeriodSeconds must cover the delay plus spring.lifecycle.timeout-per-shutdown-phase. If it does not, SIGKILL cuts in-flight checkouts.

Q5. When is blue/green a better fit than a rolling update?

When you need an atomic cut of all traffic, or a preview stack exercised by a smoke test before any learner arrives, and you can pay for two stacks. It is a poor fit when the new version’s migration is incompatible with the old, because the cut back depends on the shared database. It is unnecessary when versions are compatible and a rolling update’s partial overlap is acceptable. Blue/green is a traffic cut, not a second database.

Q6. What has to be true for a canary to be evidence?

The canary must actually receive a stated minimum of production traffic, old and new versions must both be able to handle that traffic against the same schema and topics, and an SLI you wrote in advance must be able to abort. Pod count is not a percentage of business traffic. A canary that writes a new message the old consumer cannot read is not a canary. It is data loss on five percent of orders. Health-check success is not the SLI.

Q7. How does expand/contract make an application rollback real?

You add the new column or field before you require it, you write both until old readers are gone, and you only then remove the old one. While both exist and both versions write the old one, kubectl rollout undo returns to a process that still functions. After the drop, undo brings back a process that expects a column the database no longer has. The contract step is a separate change with an inventory of every workload still on the old image, including Jobs.

Q8. A feature flag defaults to on when the flag service times out. Why is that a release bug?

Timeout becomes a silent full release of unfinished behavior, which is the opposite of a kill switch. The OpenFeature default argument should be the old, safe path. Timeouts and errors must be visible as a metric on the flag client, or you will not know you are on the default. The flag service is then not in the checkout availability path except for the cached value. A synchronous remote call with no cache makes every flag outage a checkout outage.

Q9. You undid the Deployment and the bug is still there. What are the other rollbacks?

The flag may still be on. A migration may have contracted the schema. A canary weight may still be non-zero on a Rollout that is a different object from the Deployment you undid. A ConfigMap or a topic change may have shipped beside the image. Rollback starts from an inventory of what the change included, not from the assumption that the image is the only versioned artifact. This is why the change record lists image, migrations, flags, and routes.

Q10. How would you release a rounding change in the payment case study versus a layout change in the e-commerce case study?

The layout can canary and can sit behind a release flag, with promotion on availability and on a product indicator such as payment-method selection, not only on HTTP 500s. The rounding change must not leave two regimes in one settlement period unless finance signed up for a reconciliation report. Prefer expand/contract if data shape changes, and a single flag flip or a blue/green cut at a boundary if the rule itself must be atomic. The payment design that “just canaries everything” will ship a mixed ledger. The storefront design that freezes the site for a CSS change is paying a cost the rolling update already avoided.

Hands-On Exercises

Exercise 1: Plan the cents migration

Problem. Take amount to amount_cents and add amountCents to the JSON, with 1.8 and 1.9 overlapping, and a tax flag that must be undoable without a backup.

Tasks.

1. List the ordered steps, including which step first makes rollout undo insufficient.
2. Specify maxSurge, maxUnavailable, probes, preStop, graceful shutdown, and grace period, with numbers that fit inside each other.
3. State what 1.8 does when it sees the new column and the new JSON field.
4. Define the canary or flag ramp and the PromQL or metric comparison that aborts it.
5. List the workloads you will inventory before the contract migration.
6. Say what you do if 1.9 fails readiness, and what you do if the flag’s tax path overcharges while readiness stays green.

Deliverables. The sequence, the Deployment settings, the abort rule, and the rollback matrix.

Acceptance. At no step does a Ready pod require a column or a field the other Ready version does not still write. The flag timeout defaults to the old tax path. latest does not appear.

Exercise 2: Review a “zero downtime” plan

The plan sets liveness to a query against PostgreSQL, uses image: latest, drops amount in the same Flyway file that adds amount_cents, canaries by adding one pod and promoting after five minutes with no analysis query, and turns the new tax path on whenever the flag service is unreachable. Write the review. Name the failure each choice causes during the overlap window and the replacement control.

Rubric.

Lesson Summary

A rolling update overlaps versions, so the schema and the API have to be valid for both until the old image is gone from every workload. Readiness, not liveness, gates traffic. maxUnavailable: 0 needs a surge pod. Drain is preStop, then Spring Boot’s graceful shutdown, inside a longer grace period. Blue/green cuts traffic between two stacks and still shares one database. A canary is evidence only when the new version receives real traffic and a pre-written SLI can abort it. Feature flags, evaluated through OpenFeature with a safe default, separate release from deploy and do not make a destructive migration reversible. Expand, backfill, switch reads, then contract. kubectl rollout undo is real only while the previous image can still run on the current schema. Payment rounding and a storefront layout do not share a release strategy. Module 5 should pick one per change and say what undo means after each step.

Further Reading

• Kubernetes, Deployments — maxUnavailable, maxSurge, minReadySeconds, progressDeadlineSeconds, revisionHistoryLimit, kubectl rollout undo.
• Kubernetes, Container lifecycle hooks and pod termination — preStop and terminationGracePeriodSeconds.
• Spring Boot, Graceful shutdown and the Actuator health probes — server.shutdown=graceful, /actuator/health/liveness, /actuator/health/readiness.
• Argo Rollouts, canary and blue-green — setWeight, analysis, activeService, previewService.
• OpenFeature, Java SDK — Client.getBooleanValue and EvaluationContext.
• Martin Fowler, Parallel Change and Feature Toggles — expand/contract and the four toggle kinds.
• Flyway documentation on versioned migrations — why a shipped V18 is immutable and the drop is a later version.
• Next: Enterprise Integration Patterns. A compatible rollout still has to move messages between systems that were not designed together.
• Module 5: Design a Payment Processing System and Design a Large-Scale E-Commerce Platform.