Capstone: Full Pipeline

Introduction

Capstone: full pipeline is the end-to-end design staff loops expect — one diagram from PR open to prod traffic with OIDC federation, immutable digest promotion, canary metric gates, and one-command rollback. Tools are examples; the artifact is a coherent story: triggers, stages, secrets zones, failure actions, and observability hooks on a single page.

This lesson walks through a complete fintech-grade path: PR checks → merge → build → scan → staging → canary prod → automated rollback on SLI breach.

The story

In a staff platform interview, the candidate drew three disconnected boxes — "GitHub Actions," "Kubernetes," "Datadog" — without saying what triggers deploy, what fails the pipeline, or how rollback works. The interviewer asked: "Walk me through PR #4827 from open to 5% prod traffic." The candidate could not.

The hire recommendation went to the engineer who spent 25 minutes on one diagram: PR fan-out, digest promotion, OIDC role per environment, Argo Rollouts canary with Prometheus error-rate query, and kubectl argo rollouts abort as rollback — with trade-offs stated at each gate.

Understanding the topic

The capstone integrates every prior lesson into a single delivery narrative. Non-negotiable elements: branch-triggered CI, immutable artifact, scan gate, environment promotion without rebuild, short-lived cloud credentials, progressive prod exposure, metric-gated promotion, documented rollback.

• PR path: lint + unit + SAST fan-out; no prod secrets; fork-safe.
• Merge path: build once → push digest → scan hard-fail on CRITICAL.
• Staging: OIDC deploy same digest; smoke + integration against staging dependencies.
• Prod: canary 5%→25%→100% with error-rate and latency queries; auto-abort.
• Rollback: promote previous digest from registry OR argo rollouts undo — rehearsed monthly.

Internal architecture

End-to-end capstone — PR to prod with gates:

PR opened
  ├─ lint + unit + semgrep (fan-out, no secrets)
  └─ required check → merge allowed
        ↓ merge to main
  build job → push registry/app@sha256:abc (immutable)
        ↓
  trivy scan (fail CRITICAL) + SBOM upload
        ↓ OIDC → role:deploy-staging
  Argo Rollouts → staging (100%)
  smoke: /health + payment-sandbox contract test
        ↓ manual approval OR auto if staging green 24h
  OIDC → role:deploy-prod (trust policy: main branch only)
  canary: 5% traffic → 15 min → Prometheus p99<400ms AND error<0.1%
        ↓ promote or abort
  25% → 100% OR rollback to digest sha256:prev
        ↓
  Datadog deploy marker + git SHA tag on all prod pods

Visual explanation

Two diagrams show where Full Pipeline lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Minute 0–5: clarify app (containerized API), deploy target (EKS), frequency (10×/day), SLO (99.9%).
2. Minute 5–15: draw PR → build → scan → staging → canary prod with artifacts at each hop.
3. Minute 15–22: OIDC trust per env; no AKIA keys; branch condition on prod role.
4. Minute 22–28: canary steps + Prometheus queries + abort condition.
5. Minute 28–35: rollback path, audit evidence (SBOM, approval, deploy marker), DORA metrics.

Production implementation

Capstone GitHub Actions + Argo Rollouts sketch (illustrative):

• Same digest from build through staging and prod — no rebuild per env.
• environment: production adds approval gate in GitHub if configured.
• Argo Rollouts analysis template queries Prometheus; failed analysis triggers abort.

# .github/workflows/release.yml
on:
  push: { branches: [main] }
permissions:
  id-token: write
  contents: read
jobs:
  build-scan:
    runs-on: ubuntu-latest
    outputs: { digest: ${{ steps.out.outputs.digest }} }
    steps:
      - run: docker build -t app:${{ github.sha }} . && docker push ...
      - id: out
        run: echo "digest=registry/app@sha256:..." >> $GITHUB_OUTPUT
      - run: trivy image --severity CRITICAL --exit-code 1 ${{ steps.out.outputs.digest }}

  deploy-staging:
    needs: build-scan
    environment: staging
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with: { role-to-assume: arn:aws:iam::123:role/gha-staging-deploy }
      - run: |
          kubectl argo rollouts set image payments-api \
            payments-api=${{ needs.build-scan.outputs.digest }} -n staging
          kubectl argo rollouts status payments-api -n staging --timeout 5m
      - run: curl -sf https://staging.api/health && npm run test:contract

  deploy-prod-canary:
    needs: deploy-staging
    environment: production
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with: { role-to-assume: arn:aws:iam::123:role/gha-prod-deploy }
      - run: |
          kubectl argo rollouts set image payments-api \
            payments-api=${{ needs.build-scan.outputs.digest }} -n prod
      # Rollout spec includes canary steps + analysis with Prometheus

# Rollback (documented runbook / workflow_dispatch):
# kubectl argo rollouts abort payments-api -n prod
# OR promote previous digest sha256:prev from registry

Execution workflow

Real-world use

Production capstones resemble GitHub's own deploy pipelines, Netflix Spinnaker canaries, and Google's SRE release discipline — adapted to mid-market stacks (Actions + EKS + Prometheus). Staff loops test integration thinking: you are not reciting Trivy flags; you are proving bad code cannot reach prod without passing measurable gates.

Enterprise use cases

Capstone scenario: Payments API, 10 deploys/day target, 99.95% availability, PCI scope. GitHub Enterprise + EKS + Argo CD for cluster config + Argo Rollouts for app canary + Vault for secrets not suitable for OIDC (DB creds injected at runtime).

• PR: Semgrep + unit + Checkov on IaC; merge queue ensures main always green.
• Artifact: ECR digest signed with Cosign; admission controller verifies signature at deploy.
• Prod: 5% canary 10 min, 50% 10 min, 100%; abort if 5xx rate > 0.2% for 5 min window.
• Rollback rehearsed: last 3 digests pinned in deploy workflow inputs for workflow_dispatch.
• Evidence: deploy event → SIEM with git SHA, digest, approver, canary analysis result.

Production case study

Scenario: Series B fintech, manual kubectl prod deploys, 90-min MTTR, failed staff loop twice.

• Challenge: No immutable promotion; staging ran different Dockerfile ARG than prod.
• Capstone delivered: Single workflow, digest-only promotion, OIDC, Argo canary, monthly rollback drill.
• Outcome: Change failure rate 19% → 4%; MTTR 90 → 14 min; passed staff loop on third attempt.
• Lesson: Interviewers want rollback spoken aloud — not a footnote.

Trade-offs

• Benefit: one diagram aligns eng, security, and SRE on the same truth.
• Benefit: audit and incident response trace git SHA → digest → prod pods.
• Cost: canary + analysis adds 20–40 min to prod path — worth it for payment APIs.
• Cost: OIDC + admission + signing setup is weeks of platform work upfront.
• Alternative: blue-green for simpler blast radius when metrics are immature.

Security implications

Capstone security is layered: no prod OIDC on PR, Cosign verify at admission, SBOM retained 7 years, prod role trust limited to main branch and environment protection.

• GitHub Environment protection rules: required reviewers for production job.
• IAM trust policy: sub claim matches repo + ref:refs/heads/main only.
• Secrets not in env vars — OIDC for cloud; Vault agent for DB at pod runtime.

Scalability analysis

10 deploys/day × canary analysis windows strains Prometheus query capacity. Pre-compute SLIs in recording rules; keep analysis queries simple (rate over 5m).

• Concurrency group on prod deploy — one canary at a time per service.
• Digest registry GC policy retains last 30 releases for rollback.
• Multi-region: capstone adds fan-out deploy per region with per-region analysis.

Staff engineer insights

• Lead with triggers and gates, not tool logos — "what fails the pipeline?" is question one.
• State what you defer for v1: "week one PR CI only; canary in week eight after metrics exist."
• Same digest everywhere — if you say rebuild for prod, you lose the room.
• Name DORA metric each gate improves: scan → CFR, canary → MTTR, OIDC → audit pass rate.

Best practices

• Rehearse capstone diagram monthly — team draws it from memory in 10 min.
• Link runbook rollback commands in workflow comments and PagerDuty.
• Deploy markers in observability tie incidents to git SHA within 30 seconds.
• Keep capstone on one Confluence page — single source for audits and onboarding.

Anti-patterns to avoid

• Tool salad slide — six logos, zero stage boundaries.
• Rebuild image with prod config — non-reproducible staging vs prod.
• Rollback = "revert commit and hope" — no digest promotion path.

Common mistakes

• Diagram stops at staging — interviewer asks "and then?" and time expires.
• Static AWS keys "for simplicity" on prod job — instant senior no-hire signal.
• Canary without defined query — "we'll watch dashboards" is not a gate.

Advanced interview questions

Hands-on exercise

On paper, draw the capstone for your current or last project. Annotate: 3 triggers, 5 stages, 2 OIDC roles, 1 canary query, 1 rollback command. Time yourself — 25 minutes.

• Red-team: which stage would a malicious fork PR reach?
• Identify one gate you'd soft-fail on PR but hard-fail on main — justify.

Summary

You can present an end-to-end PR→prod pipeline with OIDC, immutable digest promotion, canary metric gates, and rollback — the integration story staff platform interviews require.