Blue-Green Deployment

Introduction

Blue-green deployment runs two production-capacity environments — blue (live) and green (idle) — deploys the new version to green, validates it with smoke and integration tests at zero customer traffic, then switches the load balancer or service mesh to green in one atomic cutover. Blue stays warm for instant rollback: flip traffic back if error rates spike.

Staff engineers must also plan database migrations (both colors must read/write compatible schema), session stickiness, WebSocket draining, and the 2× infrastructure cost during the overlap window. AWS ALB target groups, Route 53 weighted records, Kubernetes with two Deployments behind one Service selector swap, and AWS CodeDeploy on ECS all implement the same pattern with different knobs.

The story

A fintech payment API used blue-green on ECS with CodeDeploy. They deployed v2.4 to green, passed synthetic checks, and switched the ALB — but v2.4 wrote a new enum value the blue database migration hadn't backfilled. Green crashed under real traffic in 90 seconds. Rollback was a target group flip back to blue in 45 seconds — saving the day — but the root cause was skipping expand-contract migration review in the blue-green checklist. They added a mandatory "schema compatibility gate" and kept blue alive for 24 hours post-cutover, accepting the cost as insurance.

Understanding the topic

Blue-green mechanics — three hard problems beyond "two environments":

• Traffic switch: ALB listener rule, Route 53 alias swap, Istio DestinationRule subset flip, or K8s Service label selector change — must be atomic and observable.
• Database migration: expand-contract pattern so blue and green run concurrently; never deploy green code that requires schema green-only until blue is retired.
• Cost: 2× compute for overlap duration; auto-scale green down after soak; finance must accept insurance cost vs outage cost trade-off.

Internal architecture

Blue-green on AWS ALB + ECS (CodeDeploy):

Internet
   ↓
Application Load Balancer
   ├─ Target Group BLUE  (task def api:42) ← 100% traffic LIVE
   └─ Target Group GREEN (task def api:43) ← 0% traffic, smoke tests
   ↓ deploy to GREEN
Run synthetic + integration on GREEN internal URL
   ↓ CodeDeploy "AllowTraffic" hook
Switch ALB weights: BLUE 0% · GREEN 100%
   ↓ soak 15–30 min (CloudWatch alarms)
Terminate BLUE tasks OR keep 24h for rollback
Rollback: CodeDeploy "StopDeployment" + flip to BLUE TG

Visual explanation

Two diagrams show where Blue-Green Deployment lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Provision green environment with identical capacity profile to blue (CPU, memory, autoscaling bounds).
2. Deploy new version to green; run smoke, contract, and load tests against green internal endpoint.
3. Verify database schema backward compatibility — both colors must operate on shared schema.
4. Execute traffic switch (ALB, DNS, or mesh); monitor error rate, latency, and business KPIs for soak window.
5. Keep blue warm for rollback SLA; terminate blue only after soak passes and change ticket closes.

Production implementation

AWS CodeDeploy blue-green ECS + appspec hooks:

• PreTrafficHook must fail the deployment if green health checks fail — do not AllowTraffic on red.
• Set `terminationWaitTimeInMinutes` to keep blue tasks for rollback (e.g. 1440 = 24h).
• CloudWatch alarms on deployment group trigger automatic rollback to blue.

# appspec.yml
version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: "<TASK_DEFINITION>"
        LoadBalancerInfo:
          ContainerName: "api"
          ContainerPort: 8080
Hooks:
  - BeforeAllowTraffic: "arn:aws:lambda:...:function:PreTrafficHook"
  - AfterAllowTraffic: "arn:aws:lambda:...:function:PostTrafficHook"

# PreTrafficHook runs synthetic checkout against GREEN TG
# buildspec deploy phase:
# aws deploy create-deployment \
#   --application-name checkout-api \
#   --deployment-group-name prod-blue-green \
#   --revision revisionType=S3,s3Location={...}

Execution workflow

Real-world use

AWS CodeDeploy popularized blue-green for enterprises on EC2 and ECS. Netflix uses red-black (same pattern, different name). Kubernetes teams implement blue-green with two Deployments and Service selector patches or Argo Rollouts blueGreen strategy. Heroku Preboot is blue-green for dynos.

Enterprise use cases

Global retailer checkout: blue-green on multi-region ALB with expand-contract PostgreSQL migrations.

• Week 1: deploy migration adding nullable column — both colors ignore it.
• Week 2: green code writes column; blue ignores — still safe.
• Week 3: blue-green cutover to green; blue retired after 48h soak.
• Cost: ~$18k/month extra ECS capacity during overlap — approved vs $2M/hour checkout outage.

Production case study

B2B SaaS API (ECS): migrated from rolling to CodeDeploy blue-green after failed rollback.

• Problem: rolling deploy mixed v1/v2 during 20-minute window; client SDK broke on response shape.
• Solution: blue-green with PreTraffic contract test suite and 24h blue standby.
• Outcome: zero mixed-version exposure; rollback tested monthly at 45 seconds via TG flip.

Trade-offs

• Benefit: instant cutover and rollback; no mixed-version requests during switch.
• Cost: double infrastructure during overlap — budget 2× for deploy window.
• Complexity: DB migrations and sticky sessions require careful expand-contract design.
• Limitation: not ideal for massive fleets where 2× pod count exceeds cluster quota.

Security implications

Green environment must receive same security posture as blue before traffic switch.

• Run container scan and WAF rule validation on green before AllowTraffic.
• Green internal URL must not bypass auth — smoke tests use production-identical IAM and mTLS.
• Secrets rotation must update both colors or use shared secret store with hot reload.
• Audit log the traffic switch event with operator identity and deployment ID.

Scalability analysis

Blue-green at scale requires capacity planning and connection management.

• Pre-warm green autoscaling before switch — cold green causes latency spike at cutover.
• Database connection pool doubling during overlap — tune max_connections on RDS.
• WebSocket and SSE clients need graceful drain on blue before switch.
• Multi-region blue-green requires coordinated DNS failover — not independent per region.

Staff engineer insights

• Blue-green without expand-contract DB migration is the #1 cause of failed cutovers — review schema in PR.
• Cost of idle green is insurance — model it as operational expense, not waste.
• Soak window length should match your incident detection MTTD — if alerts lag 15 min, soak 20 min minimum.
• Prefer CodeDeploy lifecycle hooks over manual "flip when I say so" — hooks are auditable.

Best practices

• Automate PreTraffic hooks — human smoke tests don't scale and aren't auditable.
• Keep blue tasks for at least one full business cycle before termination.
• Use CloudWatch/CodeDeploy automatic rollback alarms on 5xx rate and p99 latency.
• Document session affinity behavior — sticky clients may need drain period on blue.

Common mistakes

• Switching traffic before green passes load test at expected QPS.
• Deploying green code that depends on schema migration not yet applied to shared DB.
• Terminating blue immediately after switch — removes instant rollback capability.

Advanced interview questions

Hands-on exercise

Design blue-green for checkout API on ECS with RDS PostgreSQL:

# List PreTraffic hook checks
# Write expand-contract steps for adding payment_method column
# Define rollback trigger alarms

Summary

You understand blue-green deployment: traffic switch mechanics, database migration coupling, cost trade-offs, and production patterns with CodeDeploy, ALB, and Kubernetes — including when instant rollback beats gradual canary.