Deployment Strategies Overview

Introduction

Deployment strategies answer one question: how do we replace running software with a new version without unacceptable downtime or blast radius? Rolling updates swap pods incrementally. Blue-green maintains two full environments and switches traffic. Canary sends a small percentage of users to the new version first. Recreate tears down everything then starts fresh — fast but downtime-prone.

Staff interviews expect a comparison matrix, not a single "best" answer. The right strategy depends on statefulness, traffic shape, database migration coupling, cost tolerance, and rollback speed. Kubernetes defaults to rolling; AWS CodeDeploy excels at blue-green and canary on EC2/ECS/Lambda; service meshes (Istio, Linkerd) enable fine-grained traffic splitting without doubling infrastructure.

The story

A platform team standardized on "rolling deploy everywhere" until a stateful analytics job lost in-flight batch work during every update — `maxUnavailable: 1` killed the sole worker mid-export. Finance lost a day of reports. Another team ran blue-green for a trivial static site and doubled their AWS bill for zero benefit. The staff engineer published a decision matrix: stateless API → rolling; zero-downtime payment cutover → blue-green; high-risk algorithm change → canary with metric gates; dev/staging → recreate. Incident rate from wrong strategy choice dropped 60% in two quarters.

Understanding the topic

Four primary strategies — compare by downtime, cost, rollback speed, and operational complexity:

• Rolling: replace instances incrementally; default K8s `RollingUpdate`; minimal extra infra; slow rollback (roll forward or undo revision).
• Blue-green: two parallel stacks; instant traffic switch; 2× capacity cost during deploy; best rollback (flip back).
• Canary: progressive traffic shift (1%→100%); metric gates; smallest blast radius; needs observability and traffic control.
• Recreate: stop all old, start all new; downtime window; acceptable for dev, batch, or maintenance windows only.

Internal architecture

Strategy comparison — traffic and capacity view:

ROLLING          BLUE-GREEN         CANARY              RECREATE
[v1][v1][v1]     Blue:  v1 LIVE       95% → v1            ALL STOP
 ↓ swap 1        Green: v2 IDLE       5%  → v2            ↓ downtime
[v2][v1][v1]     flip LB ──▶ v2      gate metrics        ALL START v2
[v2][v2][v1]     Blue standby        ramp or rollback    [v2][v2][v2]
[v2][v2][v2]

Rollback: undo   Rollback: flip LB   Rollback: shift     Rollback: redeploy v1
          revision                    traffic back        (slow)

Visual explanation

Two diagrams show where Deployment Strategies Overview lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Classify the service: stateless API, stateful worker, batch, or static assets.
2. Estimate blast radius and SLO — payment tier vs internal dashboard.
3. Map strategy to tooling: K8s Deployment vs Argo Rollouts vs CodeDeploy vs Istio VirtualService.
4. Document rollback path per strategy before first prod use.
5. Review strategy quarterly — traffic and cost profiles change as services scale.

Production implementation

Same service, four implementations:

# Kubernetes — Rolling (default)
kubectl set image deploy/api api=myreg/api:v2
kubectl rollout status deploy/api

# AWS CodeDeploy — Blue-Green (appspec excerpt)
version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: arn:aws:ecs:...:task-definition/api:47
        LoadBalancerInfo:
          ContainerName: api
          ContainerPort: 8080
# DeploymentStyle: BLUE_GREEN, termination wait 30 min

# Istio — Canary traffic split
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api
spec:
  http:
  - route:
    - destination: { host: api, subset: v1 }
      weight: 90
    - destination: { host: api, subset: v2 }
      weight: 10

# Recreate — K8s strategy (dev only)
spec:
  strategy:
    type: Recreate

Execution workflow

Real-world use

Kubernetes made rolling the default for cloud-native teams. AWS CodeDeploy and Azure App Service promote blue-green for enterprises with compliance-driven rollback requirements. Google Cloud Run and Lambda use implicit rolling/recreate with platform-managed traffic. Service mesh canaries are standard at Uber and Lyft scale for algorithm changes.

Enterprise use cases

Multi-strategy platform: a large e-commerce org assigns default strategies by service tier in their internal developer portal.

• Checkout API: CodeDeploy blue-green on ECS with automatic rollback on CloudWatch alarm.
• Recommendation engine: Argo Rollouts canary with custom conversion-rate analysis.
• Product catalog: K8s rolling with 25% maxUnavailable, PDB minAvailable 2.
• Nightly ETL: recreate during maintenance window — downtime acceptable.

Production case study

Media streaming API migration: from rolling to canary after a bad codec deploy affected 30% of users.

• Incident: rolling update reached 40% of pods before elevated error rate detected; rollback took 22 minutes.
• Fix: Argo Rollouts with 2% initial canary, Prometheus buffer-underrun metric, auto-rollback.
• Result: subsequent bad release affected <0.5% sessions; rollback in 90 seconds.

Trade-offs

• Rolling pros: no extra infra, native K8s support. Cons: mixed versions during deploy; slow rollback.
• Blue-green pros: instant switch and rollback. Cons: 2× resource cost; DB migration complexity.
• Canary pros: minimal blast radius; data-driven promotion. Cons: requires metrics, traffic routing, longer deploy time.
• Recreate pros: simple, no version mixing. Cons: downtime — rarely acceptable for customer-facing prod.

Security implications

Strategy choice affects security incident blast radius during a compromised deploy.

• Canary limits exposure if a supply-chain poisoned image slips through scan gates.
• Blue-green lets you soak green with security scanners and DAST before traffic switch.
• Rolling with slow rollout gives attackers longer window on partially updated fleet — monitor anomaly detection per revision.
• Recreate creates a brief window where no healthy instances serve traffic — DDoS vulnerability during gap.

Scalability analysis

Strategy cost and complexity scale with fleet size and request volume.

• Blue-green at 500-pod scale may be prohibitively expensive — prefer canary on shared fleet.
• Rolling with low `maxUnavailable` on large Deployments extends deploy duration linearly.
• Global canary requires consistent traffic splitting at edge (CDN, GLB) not just in-cluster.
• Database connection storms happen on blue-green switch — use connection pooling and warm pools.

Staff engineer insights

• There is no universal best strategy — publish a tier matrix and enforce it in CI policy (OPA/Kyverno).
• Blue-green without DB backward-compatible migrations is a trap — schema must work on both colors.
• Canary needs a hypothesis: which metric proves success? Conversion rate, error rate, latency — pick one primary.
• Recreate in prod is a code smell unless you have a written SLO exception.

Best practices

• Default stateless HTTP services to rolling with readiness probes and PDB — simplest path.
• Require canary or blue-green in production policy for services with >99.9% SLO.
• Run load tests on green/ canary before shifting customer traffic.
• Keep a strategy decision record (ADR) when deviating from tier defaults.

Common mistakes

• Choosing blue-green for every service without calculating idle capacity cost.
• Canary without baseline metrics — you cannot detect regression vs noise.
• Rolling deploy for single-replica StatefulSets — use recreate with maintenance window instead.

Advanced interview questions

Hands-on exercise

For each service, pick a strategy and justify:

• Payment authorization API (99.99% SLO, PostgreSQL).
• Static marketing site (S3 + CloudFront).
• Nightly invoice batch job (2-hour window).
• ML ranking service (A/B sensitive to latency).

Summary

You can compare rolling, blue-green, canary, and recreate on downtime, cost, rollback speed, and complexity — and recommend the right strategy per service tier with real K8s, CodeDeploy, and mesh examples.