Rollback Strategies

Introduction

Rollback strategies restore service health after a bad deploy — faster than forward-fixing under fire. Three production paths dominate: application rollback (`kubectl rollout undo`, CodeDeploy stop deployment), traffic rollback (blue-green flip, canary weight to zero), and GitOps undo (git revert + controller sync). Staff engineers document which path applies per service, rehearse monthly, and measure MTTR — DORA's fourth key metric.

Rollback assumes you retained previous artifact revision, kept blue environment warm, or can revert git manifest to last known-good digest. Rollback without preparation is improvisation — the postmortem nobody wants.

The story

Checkout error rate spiked 400% ninety seconds after prod deploy. On-call tried `git revert` on application repo — but infra was GitOps-managed; revert did nothing until Argo CD synced. Another engineer ran `kubectl rollout undo` on the Deployment — restored v1.2.3 in 55 seconds while revert PR merged for durable fix. The team added a runbook card: "Symptom <5 min post-deploy → rollout undo first; GitOps revert for durable state; blue-green flip if on ECS." MTTR dropped from 47 minutes average to 8.

Understanding the topic

Three rollback mechanisms — know when each applies:

• Rollout undo (K8s): `kubectl rollout undo deployment/name` — reverts to previous ReplicaSet revision; fast for rolling/canary; requires revisionHistoryLimit > 0.
• Traffic rollback: blue-green ALB flip, canary abort (`kubectl argo rollouts abort`), CodeDeploy `StopDeployment` — instant for strategy-based deploys.
• GitOps undo: `git revert` manifest commit (image digest tag) + Argo CD sync — durable declared state; slower than kubectl undo but source of truth aligned.
• Forward fix: sometimes rollback impossible (DB migration irreversible) — feature flag off or hotfix forward is only path; design migrations to be reversible.

Internal architecture

Rollback decision tree:

Bad deploy detected (SLI breach)
   ↓
Within soak window?
   ├─ Blue-green active → flip traffic to BLUE (45s)
   ├─ Canary active     → argo rollouts abort (90s)
   ├─ Rolling K8s       → kubectl rollout undo (60s)
   └─ GitOps managed    → git revert manifest + argocd app sync (3–5 min)
   ↓
DB migration irreversible?
   ├─ YES → feature flag OFF + forward fix plan
   └─ NO  → confirm previous digest healthy
   ↓
Post-incident: update GitOps repo to match rolled-back state

Visual explanation

Two diagrams show where Rollback Strategies lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Detect: automated alert on error rate/latency within deploy soak window ties to recent change.
2. Triage: identify deploy mechanism (rolling, canary, blue-green, GitOps) from runbook.
3. Execute fastest safe rollback for that mechanism — undo before revert if seconds matter.
4. Verify SLI recovery; confirm traffic on known-good revision/digest.
5. Durable fix: git revert GitOps manifest to match live state; postmortem; schedule rollback drill.

Production implementation

Rollback commands by platform:

# Kubernetes rolling — immediate
kubectl rollout undo deployment/checkout-api -n prod
kubectl rollout status deployment/checkout-api -n prod

# Argo Rollouts canary
kubectl argo rollouts abort checkout-api -n prod
kubectl argo rollouts status checkout-api -n prod

# GitOps — durable state (after emergency undo)
git revert HEAD   # reverts image digest bump in manifests/repo
git push origin main
argocd app sync checkout-api --prune
argocd app wait checkout-api --health

# AWS CodeDeploy blue-green
aws deploy stop-deployment --deployment-id d-ABC123 --auto-rollback-enabled

# Verify deployed digest
kubectl get deploy checkout-api -n prod -o jsonpath='{.spec.template.spec.containers[0].image}'

Execution workflow

Real-world use

Netflix's rollback culture ("rollback first, ask questions later") influenced industry MTTR norms. Kubernetes rollout undo is the daily workhorse. GitOps teams at Weaveworks and Intuit treat git revert as authoritative rollback. AWS CodeDeploy automatic rollback on alarm is enterprise standard for ECS.

Enterprise use cases

Platform rollback runbook tiers at a global payments company:

• P0 (< 2 min): automated canary abort on CloudWatch alarm — no human.
• P1 (< 5 min): on-call runs kubectl rollout undo or TG flip from runbook link in alert.
• P2 (< 15 min): GitOps revert PR with platform approval; Argo sync.
• Monthly drill: staging bad deploy + measured rollback time logged to error budget dashboard.

Production case study

GitOps drift incident: engineer kubectl-patched prod during incident; git still declared bad digest.

• Problem: undo fixed live traffic but Argo CD auto-sync re-deployed bad version 10 min later.
• Fix: disabled auto-sync during incident, revert manifest, sync, re-enabled auto-sync.
• Policy: emergency undo allowed; mandatory git revert within 1 hour; kubectl patch prod forbidden.

Trade-offs

• kubectl undo pros: seconds-fast. Cons: drift from GitOps git if manifest not reverted.
• GitOps revert pros: durable, auditable. Cons: minutes slower; requires merge + sync.
• Traffic flip pros: instant for blue-green. Cons: only if blue still warm.
• Forward fix: only option for irreversible migration — costs more MTTR; avoid by expand-contract.

Security implications

Rollback during incident must not bypass change control entirely — document emergency actions.

• Emergency kubectl undo should trigger automated ticket and require post-incident GitOps reconcile within 24h.
• Rollback credentials (prod kubeconfig, CodeDeploy stop) limited to on-call role — not all engineers.
• Verify rolled-back image digest is not a known CVE — rollback to vulnerable version is secondary risk.

Scalability analysis

Multi-service incidents require coordinated rollback ordering.

• Rollback dependent services in reverse dependency order — API before auth if auth change broke API.
• GitOps mono-repo revert may rollback unrelated services — prefer per-app Argo Applications.
• Large ReplicaSet undo takes time proportional to pod count — set MTTR expectations accordingly.

Staff engineer insights

• Rollback runbook per service — not one generic wiki page nobody reads during incidents.
• GitOps + kubectl undo: undo first for speed, revert git within the hour for durability.
• If rollback doesn't fix SLI, bad deploy wasn't root cause — avoid rollback theater.
• Test rollback monthly in staging — untested paths fail when adrenaline is high.

Best practices

• Automate canary/blue-green rollback on SLI alarms — human rollback is backup.
• Keep revisionHistoryLimit and blue standby aligned with rollback SLA target.
• Alert annotations link directly to runbook command for that service's deploy type.
• After emergency undo, reconcile GitOps git to match cluster within SLA.

Common mistakes

• GitOps auto-sync redeploys bad manifest after manual undo — disable sync or revert git first.
• Rolling back without checking DB migration compatibility — old code on new schema breaks worse.
• Terminating blue environment before soak completes — removes fastest rollback option.

Advanced interview questions

Hands-on exercise

Write rollback runbook for service deployed via Argo Rollouts canary + GitOps manifests:

• Automated rollback trigger and command.
• Manual on-call steps in first 5 minutes.
• GitOps reconcile steps after emergency undo.
• Monthly drill success criteria.

Summary

You understand rollback strategies: kubectl rollout undo, canary abort, blue-green flip, CodeDeploy stop, and GitOps git revert — when to use each and how to reconcile speed with durable declared state.