Multi-Stage Pipelines

Introduction

Multi-stage pipelines decompose delivery into ordered, independently observable phases — build, test, scan, deploy — connected by explicit dependencies and fan-in/fan-out parallelism. Staff engineers design stages as contracts: each stage declares inputs (git SHA, prior artifacts), outputs (immutable image digest, test report, scan SBOM), and failure semantics (fail pipeline vs warn-and-continue).

This lesson teaches how to draw stage DAGs, when to parallelize vs serialize, and how fan-in gates prevent a bad artifact from reaching deploy even when individual matrix legs pass.

The story

A payments team ran one monolithic Jenkins job: compile, test, docker build, Trivy scan, and kubectl apply in a single 42-minute shell script. When unit tests failed at minute 18, the job still built and pushed an image because the script used set +e on the test step. A hotfix merged Friday at 4 p.m. reached prod with a failing test suite — caught only when settlement reconciliation broke Saturday morning.

The postmortem split the job into four stages with needs: dependencies. Build publishes an artifact; test consumes it; scan consumes the image digest; deploy requires all three green. Fan-in on deploy blocked 11 bad releases in the next quarter before they touched staging.

Understanding the topic

A stage is a bounded unit of work with a single accountability owner, measurable SLA, and clear pass/fail criteria. Fan-out runs independent work in parallel (unit + lint + SAST on PR). Fan-in aggregates results before a privileged stage (deploy waits for test + scan + approval).

• Build stage: produces immutable artifact tagged with git SHA; no environment-specific config baked in.
• Test stage: consumes artifact or source; outputs junit/coverage; may fan-out into integration + contract legs.
• Scan stage: consumes image or lockfile; outputs SBOM + CVE report; warn vs fail policy per severity.
• Deploy stage: promotes digest, not tag; requires fan-in from test + scan + optional manual approval.
• Failure semantics: define per stage — hard fail blocks downstream; soft fail logs ticket but continues.

Internal architecture

PR pipeline DAG — fan-out after build, fan-in before deploy:

push / pull_request
   ↓
┌─ build (compile, package, push image:sha)
│     ↓ fan-out
│  ├─ unit-tests ─────┐
│  ├─ integration ───┤ fan-in
│  └─ sast-scan ──────┤
│     ↓               ↓
│  contract-tests ────┤
│     ↓               ↓
└─ scan-image (Trivy on digest) ──┐
                                   ↓ all green
                            deploy-staging (OIDC)
                                   ↓ smoke + metrics
                            deploy-prod (canary gate)

Visual explanation

Two diagrams show where Multi-Stage Pipelines lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. List stages left-to-right: trigger → build → verify → scan → deploy → observe.
2. Assign each stage one owner team and one SLI (duration p95, pass rate).
3. Draw fan-out legs that share no secrets with untrusted PR forks.
4. Add fan-in job that blocks deploy unless all upstream stages succeeded.
5. Document rollback: which stage artifact to promote N-1 from registry.

Production implementation

GitHub Actions multi-stage with explicit needs and artifact handoff:

• needs: creates the DAG — deploy cannot start until fan-in completes.
• Outputs pass digest between jobs; never rebuild image in deploy stage.
• Separate jobs isolate secrets: scan job has no kubectl credentials.

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      image: ${{ steps.meta.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - id: meta
        run: |
          docker build -t app:${{ github.sha }} .
          docker push registry/app:${{ github.sha }}
          echo "digest=registry/app@sha256:abc..." >> $GITHUB_OUTPUT

  unit:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: npm ci && npm test -- --reporter=junit

  integration:
    needs: build
    runs-on: ubuntu-latest
    services:
      postgres: { image: postgres:16 }
    steps:
      - run: npm run test:integration

  scan:
    needs: build
    steps:
      - run: trivy image --severity HIGH,CRITICAL --exit-code 1 ${{ needs.build.outputs.image }}

  deploy-staging:
    needs: [unit, integration, scan]
    environment: staging
    permissions:
      id-token: write
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with: { role-to-assume: arn:aws:iam::123:role/gha-staging }

Execution workflow

Real-world use

Google's TAP, Meta's stacked CI, and Spotify's Backstage deployment plugins all enforce multi-stage DAGs. GitHub Actions needs, GitLab stages, Azure stages, and Jenkins parallel + input steps express the same idea. DORA high performers average <15 min from commit to staging — achievable only when stages run in parallel with tight fan-in gates.

Enterprise use cases

A 120-engineer B2B SaaS on GitLab CI uses parent pipeline → child pipelines per service, with a final fan-in release-coordinator stage that verifies all service digests before Argo CD sync.

• Build child pipelines trigger on path filters — only affected services rebuild.
• Scan stage uploads CycloneDX SBOM to artifact registry; SOC2 auditors query by release tag.
• Deploy stage is GitOps-only: pipeline updates image digest in git; Argo syncs cluster.
• Failed fan-in leg blocks the release train; Slack bot posts which stage/owner failed.

Production case study

Scenario: Neobank with 35 microservices, GitHub Actions, EKS. Monolithic workflow averaged 38 min; deploy happened even when contract tests were skipped manually.

• Challenge: 14% change failure rate; auditors could not map git SHA to scan evidence.
• Decision: Split into build / test-matrix / scan / deploy-staging / deploy-prod with digest fan-in.
• Outcome: PR p95 dropped to 11 min; change failure rate fell to 5%; SOC2 evidence automated.
• Lesson: fan-in deploy job is the compliance gate — design it before optimizing build cache.

Trade-offs

• Benefit: blast-radius containment — a bad test never reaches deploy.
• Benefit: parallel fan-out cuts wall-clock; p95 PR feedback drops sharply.
• Cost: more YAML surface area; stage boundaries need ongoing ownership.
• Cost: artifact storage and cross-job handoff add platform complexity.
• Risk if skipped: monolithic jobs hide failures and mix untrusted PR code with prod credentials.

Security implications

Stage boundaries are security boundaries. Untrusted fork PRs should only reach build + unit stages on isolated runners — never scan-with-registry-write or deploy stages.

• Use pull_request_target only with extreme caution; prefer pull_request + restricted secrets.
• Deploy stages get OIDC roles; build stages get read-only registry push scoped to branch.
• Scan stage outputs SBOM for supply-chain audit; tie CVE policy to fail-before-deploy.

Scalability analysis

At 200+ PRs/day, stage queue depth and runner pool sizing dominate. Fan-out multiplies runner minutes — budget per stage and enforce concurrency groups on deploy.

• concurrency: group: deploy-prod, cancel-in-progress: false prevents overlapping prod pushes.
• Remote build cache at build stage amortizes cost across fan-out test legs.
• Stage duration dashboards per team — identify which leg blocks the fan-in gate.

Staff engineer insights

• Draw the DAG on paper before writing YAML — interviewers reward stage diagrams over tool names.
• Name one metric per stage; if you cannot measure it, the stage is not real.
• Soft-fail scan on PR, hard-fail on main — tune policy by branch, not globally.
• Deploy stage promotes digest, never rebuilds — rebuilding in deploy is the #1 reproducibility bug.

Best practices

• One artifact per build; downstream stages consume, never rebuild.
• Publish junit, coverage, and SBOM as stage outputs for audit and dashboards.
• Keep deploy stages idempotent — same digest applied twice is safe.
• Version stage templates; org-wide golden path via reusable workflows.

Anti-patterns to avoid

• "One shell script to rule them all" — hides which step failed and who owns it.
• Deploy on green build only — skipping scan and integration tests.
• Re-running deploy without re-running verification after a manual fix.

Common mistakes

• Using continue-on-error: true on test stages without a fan-in blocker.
• Mixing staging and prod credentials in the same job matrix.
• Fan-out without concurrency limits — 12 legs × 200 PRs/day exhausts runner quota.

Advanced interview questions

Hands-on exercise

Sketch a DAG for a Node.js API with Docker deploy to EKS. Include: PR fan-out (unit, eslint, npm audit), image scan, staging deploy with smoke test, prod canary requiring manual approval.

• Label each edge with the artifact passed (SHA, digest, junit URL).
• Mark which jobs need OIDC vs read-only tokens.
• Identify one soft-fail vs hard-fail policy decision and justify it.

Summary

You can decompose delivery into build → test → scan → deploy stages, wire fan-out parallelism and fan-in gates, and explain why stage boundaries map to security zones and audit evidence. Teach the DAG back without notes.