Rolling Deployments

Introduction

Rolling deployments replace running instances incrementally — old pods terminate as new pods become ready — without provisioning a second full environment. Kubernetes RollingUpdate is the default: controlled by maxSurge (how many extra pods during update) and maxUnavailable (how many pods can be down). Health probes (readinessProbe, livenessProbe, startupProbe) determine when a new pod receives traffic.

Staff engineers tune rolling parameters against PodDisruptionBudgets, HPA behavior, and connection draining. Rolling is the right default for stateless HTTP services — wrong for single-replica workers, stateful monoliths without probes, or changes requiring atomic cutover.

The story

An API Deployment used default rolling settings on a 3-replica service during a bad release — `maxUnavailable: 25%` allowed one pod down while new pods crash-looped on a missing env var. Effective capacity dropped to 2/3 then 1/3; circuit breakers opened upstream. Fix: readinessProbe hitting `/ready` (not just `/health`), `maxUnavailable: 0` with `maxSurge: 1` for zero-downtime roll, PDB `minAvailable: 2`, and preStop hook with 15s sleep for connection drain. Next bad release: kubectl rollout undo restored stable in 40 seconds with zero customer-visible errors.

Understanding the topic

Kubernetes rolling update controls:

• maxUnavailable: max pods that can be unavailable during update (absolute or percent) — lower = safer, slower deploy.
• maxSurge: max extra pods above desired count — enables zero-downtime when maxUnavailable=0 and maxSurge≥1.
• readinessProbe: pod receives Service endpoints only when ready — prevents traffic to starting/c broken pods.
• livenessProbe: restarts hung containers; misconfigured liveness kills pods mid-deploy.
• PDB minAvailable: blocks eviction below minimum — protects rolling from taking too many down.

Internal architecture

Rolling update sequence (replicas=4, maxSurge=1, maxUnavailable=0):

t0: [v1][v1][v1][v1]           desired=4, all ready
t1: [v1][v1][v1][v1][v2↑]       surge v2 pod starting
t2: [v1][v1][v1][v1][v2✓]       v2 passes readiness → EP added
t3: [v1][v1][v1][v2✓]           v1 pod terminated (maxUnavailable=0 maintained)
... repeat until all v2
kubectl rollout status deployment/api
kubectl rollout undo deployment/api  # rollback to previous RS

Visual explanation

Two diagrams show where Rolling Deployments lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Configure readinessProbe with meaningful check (DB ping, dependency OK) — not static 200 OK.
2. Set maxUnavailable: 0 and maxSurge: 25% for zero-downtime on critical services.
3. Add PDB minAvailable aligned with replica count and SLO.
4. Add preStop lifecycle hook (sleep + graceful shutdown) for connection draining.
5. Monitor rollout: kubectl rollout status; rollback with kubectl rollout undo on elevated errors.

Production implementation

Production Deployment spec with probes and rolling strategy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: api
        image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/checkout-api:v2.3.1
        readinessProbe:
          httpGet: { path: /ready, port: 8080 }
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet: { path: /healthz, port: 8080 }
          initialDelaySeconds: 30
        startupProbe:
          httpGet: { path: /healthz, port: 8080 }
          failureThreshold: 30
          periodSeconds: 5
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api-pdb
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: checkout-api

Execution workflow

Real-world use

Rolling is the Kubernetes default used by most cloud-native teams daily. Helm and Kustomize encode rolling parameters in golden paths. VM-based rolling exists in AWS ASG instance refresh and Azure VMSS rolling upgrades — same incremental replacement idea with different health signals.

Enterprise use cases

Platform golden path: internal Helm chart enforces rolling defaults by tier.

• Tier 1: maxUnavailable 0, maxSurge 25%, PDB minAvailable 75%, mandatory readiness on /ready.
• Tier 2: maxUnavailable 1, maxSurge 1, PDB minAvailable 50%.
• CI policy: Kyverno rejects Deployments without readinessProbe in prod namespaces.

Production case study

High-traffic gateway (80 replicas): tuned rolling after latency incident.

• Issue: maxUnavailable 25% removed 20 pods simultaneously during peak — p99 spiked 3×.
• Fix: maxUnavailable 0, maxSurge 10%, PDB minAvailable 70%, preStop drain.
• Result: deploy duration increased 8 min but zero latency spikes during rollouts.

Trade-offs

• Benefit: no 2× infrastructure cost; native K8s; simple mental model.
• Cons: mixed v1/v2 during rollout — API compatibility required; slower rollback than blue-green flip.
• Probe risk: bad readiness blocks rollout; bad liveness causes restart storms.
• Stateful: StatefulSet rolling order matters — pod-0 before pod-1 for quorum systems.

Security implications

Rolling exposes mixed versions — security patch rollout must ensure all pods reach patched revision.

• Verify rollout completion: kubectl rollout status — partial roll leaves vulnerable pods.
• Image pull policy Always — rolling with :latest tag causes non-deterministic fleet.
• NetworkPolicy must apply to new pods before readiness adds them to Service endpoints.

Scalability analysis

Large Deployments with conservative maxUnavailable extend deploy duration.

• 100 replicas with maxSurge 1 → ~100 sequential waves — consider maxSurge 10% with PDB.
• HPA scaling during rollout can fight maxSurge — pause HPA or use rollout pause annotation.
• Long startup times require startupProbe — otherwise liveness kills slow-start containers.

Staff engineer insights

• readinessProbe must check dependencies — /health returning 200 while DB is down poisons the rollout.
• maxUnavailable: 0 + maxSurge ≥ 1 is the zero-downtime rolling recipe — memorize it.
• kubectl rollout undo is fast rollback — but only if previous ReplicaSet is retained (default 10 revisions).
• StatefulSet rolling ≠ Deployment rolling — understand pod management policy before touching quorum services.

Best practices

• Use maxUnavailable: 0 for customer-facing services — surge new pods before killing old.
• startupProbe protects slow-init containers from premature liveness kills.
• preStop sleep allows load balancers to deregister pod before SIGTERM processing.
• Keep revisionHistoryLimit ≥ 5 for rollback headroom.

Common mistakes

• Readiness probe too shallow — passes while app cannot serve real traffic.
• maxUnavailable 25% on small replica counts (3 pods) — allows 1 down, risky at 2/3 capacity.
• Rolling update on single-replica Deployment — use recreate or accept brief downtime.

Advanced interview questions

Hands-on exercise

Tune a Deployment with 8 replicas for 99.9% availability:

• Set maxSurge and maxUnavailable with justification.
• Write readinessProbe that checks PostgreSQL connectivity.
• Define PDB minAvailable for at most 1 pod down during voluntary disruption.

Summary

You understand rolling deployments: Kubernetes maxSurge/maxUnavailable, readiness/liveness/startup probes, PDB interaction, and production tuning for zero-downtime rollouts.