Continuous Deployment

Introduction

Continuous Deployment (CDep) means every merge to main that passes CI gates ships to production automatically — no Friday deploy calendar, no hero engineer clicking "Approve." It is the highest maturity step on the DORA ladder: Continuous Integration verifies every change; Continuous Delivery keeps prod releasable; Continuous Deployment removes the human approval gate entirely.

Staff engineers treat CDep as a system design problem, not a checkbox. You need stable CI, immutable artifacts, deploy strategies matched to blast radius (rolling, blue-green, canary), feature flags for dark launches, metric gates that halt bad releases, and rehearsed rollback paths. Progressive delivery — combining canary analysis with automated promotion or rollback — is how Netflix and GitHub ship dozens of times daily without betting the company on every commit.

The story

A payments API team enabled "continuous deployment" by wiring GitHub Actions to `kubectl apply` on every green main build. Within a week they shipped a schema migration that passed unit tests but broke checkout for 12% of traffic — no canary, no feature flag, no error-rate gate. MTTR was four hours because nobody knew whether to `git revert`, `kubectl rollout undo`, or flip an ALB. The postmortem rebuilt CDep properly: trunk-based flow, Trivy scan gate, Argo Rollouts canary with Prometheus error-rate analysis, LaunchDarkly flags for the new pricing path, and a runbook where rollback meant "halt canary + revert flag" in under three minutes.

Understanding the topic

Continuous Deployment automates the last mile — but only safely when upstream gates and downstream deploy mechanics are designed together:

• Auto-deploy gates: CI must fail on lint, unit, integration, SAST, and container scan before any deploy job runs; optional staging soak with synthetic checks.
• Deploy strategy (blast radius): rolling for stateless K8s, blue-green for zero-downtime cutover, canary for metric-validated progressive delivery — pick per service tier.
• Feature flags: decouple deploy from release; ship code dark, enable for internal users, then ramp percentage without redeploying.
• Progressive delivery: Argo Rollouts, Flagger, or CodeDeploy canary — automated promotion when SLIs pass, rollback when error budget burns.
• Rollback: `kubectl rollout undo`, ALB target flip, git revert + GitOps sync — rehearsed monthly, not invented during incidents.

Internal architecture

Continuous Deployment architecture — git merge triggers a gated pipeline that promotes one immutable artifact through progressive delivery:

merge → main (trunk)
   ↓
CI gates: test · lint · SAST · Trivy scan
   ↓
Build + push image (digest sha256:abc…)
   ↓
Deploy staging (smoke + contract tests)
   ↓
CDep gate: metric baseline OK?
   ↓
Progressive prod: canary 5% → 25% → 100%
   ├─ feature flag OFF (dark deploy)
   ├─ SLI gate: error_rate < 0.1%
   └─ auto-rollback on breach
   ↓
Observe · DORA metrics · audit trail

Visual explanation

Two diagrams show where Continuous Deployment lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Stabilize CI on every PR — no CDep until main is green 95%+ of the time.
2. Build once, tag image with git SHA and push to registry with digest recorded in SBOM.
3. Deploy to staging automatically; run smoke, contract, and migration dry-run tests.
4. Promote same digest to prod via canary (Argo Rollouts or CodeDeploy) with Prometheus/Datadog gates.
5. Enable feature flag for internal cohort first; ramp traffic only when SLIs hold for the soak window.

Production implementation

GitHub Actions + Argo Rollouts canary with metric analysis:

• Gate deploy job on `needs: [test, scan]` — never deploy from a red pipeline.
• Use GitHub `environment: production` with required reviewers only until canary gates are trusted.
• Record deployed digest in deployment annotation for audit: `kubectl annotate deploy … git-sha=${{ github.sha }}`.

# .github/workflows/deploy-prod.yml
name: Deploy Prod
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-deploy-prod
          aws-region: us-east-1
      - run: |
          IMAGE=123456789012.dkr.ecr.us-east-1.amazonaws.com/checkout-api:${{ github.sha }}
          kubectl argo rollouts set image checkout-api checkout-api=$IMAGE -n prod
          kubectl argo rollouts promote checkout-api -n prod --full=false
      - name: Wait for canary analysis
        run: |
          kubectl argo rollouts status checkout-api -n prod --timeout 15m
# Rollout spec excerpt (analysis template references Prometheus)
# spec:
#   strategy:
#     canary:
#       steps: [{ setWeight: 5 }, { pause: { duration: 5m } }, { setWeight: 25 }]
#       analysis:
#         templates:
#         - templateName: error-rate-check

Execution workflow

Real-world use

GitHub, GitLab, and Shopify publish engineering blogs on shipping main to prod daily. DORA "elite" performers deploy multiple times per day with change failure rates under 15%. The pattern is always the same: strong CI, immutable artifacts, progressive delivery, and feature flags — not faster SSH.

Enterprise use cases

Etsy-scale CDep: trunk-based development, feature flags for every user-visible change, and deploy strategies tiered by service criticality.

• Tier 1 (payments): canary + manual approval on final 100% until error budget proves automation.
• Tier 2 (catalog): rolling update with HPA and readiness probes; flags for A/B experiments.
• Tier 3 (internal tools): direct rolling deploy after CI green — acceptable blast radius.
• Culture: "If you can't roll back in five minutes, you can't auto-deploy."

Production case study

SaaS billing team (40 engineers): moved from bi-weekly manual deploys to CDep with canary and flags.

• Before: 4-hour deploy window, 18% change failure rate, MTTR 90 minutes.
• Changes: trunk-based flow, Argo Rollouts canary, LaunchDarkly flags, Trivy gate, OIDC deploy role.
• After 6 months: daily deploys, 4% change failure rate, MTTR 12 minutes, SOC2 audit passed with pipeline evidence.
• Key lesson: they deferred full auto-promote until canary rollback succeeded in three staging fire drills.

Trade-offs

• Benefit: lead time from merge to prod drops from days to minutes; engineers get fast feedback on real traffic.
• Cost: upfront investment in test stability, observability, flags, and deploy automation.
• Risk: auto-deploy without gates ships defects faster — CDep amplifies both good and bad discipline.
• Org trade-off: requires trunk-based flow and shared ownership of prod health, not a separate "release team."

Security implications

CDep expands the attack surface of your pipeline — prod credentials and deploy keys must never be available to untrusted PR workflows.

• Use OIDC to assume short-lived IAM roles; block `AKIA…` long-lived keys in repo secrets.
• Separate runner pools: PR jobs cannot reach prod clusters even if YAML is maliciously edited.
• Sign artifacts (cosign) and verify at admission — prevent registry swap attacks mid-pipeline.
• Audit every auto-deploy: who merged, which digest landed, which gates passed.

Scalability analysis

High-frequency CDep stresses registries, clusters, and observability backends.

• Rate-limit concurrent rollouts per namespace to avoid etcd and API server thundering herds.
• Shard CI with path filters or monorepo affected detection so every commit doesn't rebuild the world.
• Pre-warm canary pods before traffic shift — cold start latency skews SLI comparisons.
• Centralize deploy metrics (deployment frequency, failed canaries) for platform SLO dashboards.

Staff engineer insights

• CDep is a consequence of CI maturity — measure flaky test rate before removing human prod approval.
• Feature flags are not optional for user-visible changes; they decouple deploy risk from product risk.
• Name three rollback paths in your design doc: Kubernetes undo, traffic switch, git revert — pick per service.
• Progressive delivery (canary + analysis) is how staff engineers justify CDep to risk-averse leadership.

Best practices

• Promote the same container digest from staging to prod — never rebuild per environment.
• Tag every prod deploy with git SHA, pipeline run ID, and artifact digest in Kubernetes annotations.
• Start with auto-deploy to staging only; add prod automation after rollback drills succeed.
• Pair CDep with error budgets — pause auto-promote when SLO burn rate exceeds threshold.

Common mistakes

• Enabling prod auto-deploy before integration and contract tests cover critical paths.
• Using rolling deploy for stateful services without understanding pod disruption budgets.
• Treating feature flags as permanent if-branches — schedule flag retirement in tickets.

Advanced interview questions

Hands-on exercise

Design a CDep pipeline for an API with 99.9% availability SLO:

• List CI gates that must pass before any deploy job.
• Choose deploy strategy and justify blast radius.
• Define two SLI queries that halt canary promotion.
• Write rollback steps for a bad canary at 25% traffic.

Summary

You understand Continuous Deployment as a gated system: stable CI, immutable artifacts, deploy strategies matched to blast radius, feature flags for dark launches, metric-driven progressive delivery, and rehearsed rollback. Explain how blue-green, canary, and rolling fit under one CDep program without reading notes.