Mock: Pipeline Design

Introduction

Mock: pipeline design — full staff-loop simulation. Scenario: Series C fintech, 50 engineers, 12 microservices on EKS, SOC2 Type II audit in 6 months, trunk-based development, target 5× daily deploys per service. Design the org pipeline from PR to prod including compliance evidence, secrets, and governance — in 40 minutes, out loud.

The story

This is the interview room on a Tuesday. The hiring manager says: "We're LedgerWave — payments and ledger APIs, SOC2, 50 engineers, everything on EKS. Our change failure rate is 14%. Design how CI/CD should work." You have the whiteboard. The framework lesson is your spine; this mock is the graded exam with full rubric and model answer.

Understanding the topic

LedgerWave constraints (given — do not skip): 50 eng / 12 services / EKS / SOC2 / trunk-based / 5 deploys per service per day / current CFR 14% / no long-lived cloud keys in git / auditors need automated test + scan evidence per prod digest.

• Org shape: 4 product squads + 1 platform squad (6 eng) owning golden paths.
• Compliance: SOC2 CC8 change management — prove test + scan before prod; retain 1 year.
• Tech: GitHub Enterprise, ECR, Argo CD (cluster config), Argo Rollouts (app canary), Vault for app secrets.
• Pain today: copy-paste workflows, manual prod approval via Slack, staging ≠ prod config.
• Success metrics: CFR <5%, lead time <1 hr to staging, audit zero findings on CC8.

Internal architecture

Model answer — LedgerWave pipeline (whiteboard target):

PR (fork-safe)
  paths-filter → affected services only
  fan-out: unit ∥ eslint ∥ semgrep ∥ checkov(IaC)
  contract-tests (Pact) against staging mocks
  merge queue on main (required checks)

merge → main
  reusable: org/container-release@v3 (build + Trivy + Cosign sign + SBOM→S3)
  output: digest + sbom-url
  fan-in: all affected services built

deploy-staging (OIDC gha-staging-role)
  Argo Rollouts 100% per service
  smoke + Pact verify + synthetic payment $0.01

deploy-prod (environment protection + optional auto)
  Argo Rollouts canary 5→50→100
  analysis: 5xx <0.1%, p99<300ms (Prometheus)
  abort → auto rollback stable digest

governance
  org ruleset: must use container-release@v3
  evidence: SHA + digest + SBOM + approver → SIEM
  rollback: workflow_dispatch previous digest (last 5 retained)

Visual explanation

Two diagrams show where Pipeline Design lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Minute 0–5: repeat constraints aloud; confirm trunk-based + 12 services + SOC2 CC8.
2. Minute 5–15: monorepo path filters; affected service matrix; merge queue; no prod secrets on PR.
3. Minute 15–25: org reusable container-release; Cosign + SBOM; OIDC staging per service.
4. Minute 25–33: prod canary + environment protection; evidence pipeline to SIEM.
5. Minute 33–40: phased plan weeks 1–12; rollback; defer multi-region until Q2.

Production implementation

LedgerWave service caller workflow (excerpt):

• Path filters prevent 12-service full rebuild on every PR.
• Reusable workflows enforce SOC2 scan+sign policy org-wide.
• Per-service production environment enables squad-level approval.

# services/ledger-api/.github/workflows/ci.yml
on:
  pull_request:
    paths: ['services/ledger-api/**', 'libs/ledger-core/**']
  push:
    branches: [main]
    paths: ['services/ledger-api/**', 'libs/ledger-core/**']

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs: { affected: ${{ steps.f.outputs.ledger }} }
    steps:
      - uses: dorny/paths-filter@v3
        id: f
        with: { filters: 'ledger=true
services/ledger-api/**' }

  pr-checks:
    if: github.event_name == 'pull_request'
    needs: changes
    strategy: { matrix: { check: [unit, lint, semgrep, pact] } }
    runs-on: ubuntu-latest
    steps:
      - run: npm test / eslint / semgrep --config org-rules

  release:
    if: github.ref == 'refs/heads/main'
    uses: ledgerwave/.github/.github/workflows/container-release@v3
    with: { service: ledger-api, sign: true }
    secrets: inherit

  deploy-staging:
    needs: release
    uses: ledgerwave/.github/.github/workflows/deploy-rollout@v2
    with: { digest: ${{ needs.release.outputs.digest }}, env: staging }

  deploy-prod:
    needs: deploy-staging
    environment: production-ledger-api
    uses: ledgerwave/.github/.github/workflows/deploy-rollout@v2
    with: { digest: ${{ needs.release.outputs.digest }}, env: prod, strategy: canary }

Execution workflow

Real-world use

LedgerWave is a composite of real fintech migrations: trunk-based + merge queue (Shopify), golden paths (GitHub org rulesets), Cosign (Sigstore adoption), Pact (microservices contract testing), SOC2 CC8 (Vanta/Drata control libraries). Your model answer should feel borrowable Monday morning.

Enterprise use cases

Rubric — how interviewer scores LedgerWave mock (100 pts):

• 20 pts — Clarifying questions and constraint playback.
• 25 pts — CI DAG with fork safety, path filters, merge queue.
• 20 pts — Immutable digest + scan + sign + SBOM evidence chain.
• 20 pts — OIDC staging/prod + canary + rollback spoken.
• 15 pts — Phased rollout + trade-offs (what deferred, why).
• Pass threshold: 75+ with no critical miss (prod secrets on PR, no rollback, rebuild per env).

Production case study

LedgerWave phased rollout (model answer close):

• Weeks 1–3: PR CI + path filters + branch protection; no CD yet.
• Weeks 4–6: container-release@v1 (build+scan); manual staging deploy.
• Weeks 7–9: OIDC staging auto + Cosign + SBOM to S3; SIEM integration.
• Weeks 10–12: prod canary one pilot service; expand after CFR drops.
• Audit: demo trace digest→SBOM→test report→approver for auditor walkthrough.

Trade-offs

• Chosen: reusable golden path — SOC2 consistency over squad autonomy.
• Chosen: canary per service — CFR reduction worth 30 min prod latency.
• Deferred: multi-region active-active — Q2 after single-region CFR <5%.
• Deferred: developer self-service prod deploy — environment protection stays until CFR stable 2 quarters.
• Rejected: per-squad custom deploy YAML — audit failure risk too high.

Security implications

LedgerWave SOC2 requires proving unauthorized code cannot reach prod. Model answer: Cosign admission, OIDC branch-scoped, SBOM retained, no pull_request_target with write creds.

• Semgrep org rules block hardcoded PAN patterns in PR CI.
• Checkov on Terraform modules in same PR pipeline.
• Vault Agent Injector for DB creds — not CI env vars.

Scalability analysis

50 eng × 5 deploys/day × 12 services = 300 deploy events/day peak. Merge queue batches main commits; Argo CD sync concurrency limited per cluster; ECR lifecycle retains 50 digests/service.

• Platform squad owns callee on-call — product squads don't fork workflows.
• Pact broker scales contract verification — block deploy if consumer contract broken.
• Prometheus recording rules pre-aggregate canary queries — avoid analysis timeout.

Staff engineer insights

• Say "SOC2 CC8" by name — shows you know what auditors actually test.
• 50 engineers implies merge queue — say it before interviewer prompts.
• 12 services implies path filters and reusable workflows — not 12 bespoke pipelines.
• Close with pilot service name ("start with ledger-api, highest CFR") — concrete beats abstract.

Best practices

• Name pilot service and why — shows prioritization.
• Quantify current CFR and target — interviewers love numbers.
• Map every control to SOC2 CC8 sub-requirement.
• Platform squad ownership model — who maintains golden path.

Anti-patterns to avoid

• Slack approval as permanent prod gate — mention migrating to environment protection.
• Staging SSH deploy outside pipeline — contradicts SOC2 story.
• Static AWS keys "for now" — critical rubric fail.

Common mistakes

• Designing 12 identical full pipelines — miss path filters and reuse.
• Ignoring merge queue at 50 engineers on trunk-based.
• SOC2 "we'll document manually" — automatic evidence is the answer.

Advanced interview questions

Hands-on exercise

Run LedgerWave mock solo: 40-minute timer, speak aloud, score with rubric. Then mutate scenario: HIPAA instead of SOC2, 8 engineers instead of 50 — adapt in 15 minutes.

• Write phased 12-week plan on paper after mock.
• List 3 questions you'd ask LedgerWave PM that change the design.

Summary

You can deliver a 40-minute LedgerWave pipeline design — fintech, 50 eng, SOC2, K8s — with golden paths, compliance evidence, canary prod, and phased rollout.