Mock: Pipeline Incident

Introduction

Mock: pipeline incident — narrate a bad deploy Friday 4:47 p.m. through detection, mitigation, and postmortem pipeline fixes. Scenario: Checkout API v2.14.0 reaches 100% prod traffic; error rate jumps 0.05% → 4.2%; revenue loss $38k/hour. The pipeline both failed you (how) and saved you (how). Staff loops use this to test operational thinking, not calm theater.

The story

Friday 4:47 p.m. — deploy pipeline green. 4:52 p.m. — PagerDuty fires "checkout 5xx SLO breach." On-call opens Grafana: spike aligns with Argo Rollout full promotion at 4:51. v2.14.0 digest sha256:bad99. Rollback button exists. Incident lasts 23 minutes. Monday postmortem asks: why did CI not catch it; why wasn't canary stopped at 4%; what pipeline changes prevent recurrence? This mock is that postmortem presentation.

Understanding the topic

Incidents reveal pipeline gaps and pipeline saves. Gaps: missing contract test, canary analysis used wrong metric, staging lacked prod feature flag state. Saves: immutable digest enabled exact rollback; deploy marker correlated SHA to metric; previous digest still in registry; OIDC meant no human SSH scramble.

• T+0 detection: canary analysis should have aborted at 5% — why didn't it?
• T+5 triage: correlate deploy marker, digest, git SHA, config diff.
• T+8 mitigate: argo rollouts abort OR promote sha256:good88 digest.
• T+23 resolved: error rate normal; customers compensated; incident doc opened.
• T+72 postmortem: pipeline action items with owners — not "be more careful."

Internal architecture

Incident timeline vs pipeline stages:

4:30  CI green — unit passed, integration passed, scan WARN (not fail)
4:35  Staging deploy digest sha256:bad99 — smoke passed
4:40  Prod canary 5% → analysis query used avg latency not p99
4:45  Auto-promote to 100% (analysis false negative)
4:51  5xx rate 4.2% — SLO burn
4:53  On-call: argo rollouts abort → traffic to sha256:good88
4:56  Error rate 0.06% — stable
5:10  Post-incident: disable auto-promote; fix analysis template

PIPELINE SAVED:
• Rollback = previous digest (30 sec)
• Deploy marker → instant SHA correlation
• No rebuild — promoted good88 from registry

PIPELINE FAILED:
• WARN-only CVE + skipped contract test on main
• Canary analysis metric misconfigured
• Staging missing LaunchDarkly prod flag parity

Visual explanation

Two diagrams show where Pipeline Incident lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Draw timeline on whiteboard — deploy events aligned to metric graph.
2. Identify detection gap: why canary didn't abort (wrong PromQL query).
3. Identify prevention gap: contract test skipped due to paths-filter bug.
4. State mitigation: rollback via digest, not git revert + rebuild (13 min saved).
5. Close with 3 pipeline action items: fail contract on main, fix analysis, staging flag sync.

Production implementation

Broken vs fixed canary analysis template (root cause excerpt):

• Canary must query canary-specific labels — not cluster-wide averages.
• Contract tests on main before deploy — PR-only tests miss merge regressions.
• WARN scan on CRITICAL CVE allowed bad base image — changed to fail.

# BROKEN — avg latency hides tail 5xx errors
metrics:
  - name: latency-ok
    successCondition: result < 500   # avg ms — WRONG
    provider: prometheus
    query: avg(http_request_duration_ms)

# FIXED — error rate gate on canary
metrics:
  - name: error-rate
    successCondition: result < 0.001
    provider: prometheus
    query: |
      sum(rate(http_requests_total{status=~"5..",rollout_hash="{{args.canary-hash}}"}[2m]))
      / sum(rate(http_requests_total{rollout_hash="{{args.canary-hash}}"}[2m]))
  - name: p99-latency
    successCondition: result < 400
    query: histogram_quantile(0.99, ...)

# Postmortem action — contract test no longer optional
jobs:
  contract:
    if: github.ref == 'refs/heads/main'   # was: only on PR
    steps:
      - run: npm run pact:verify -- --fail-fast

Execution workflow

Real-world use

Real incidents (Knight Capital, GitLab 2017 delete prod, numerous payment outages) share patterns: green CI, fast prod promotion, slow rollback. Modern pipelines save teams with digest rollback and GitOps revert; they fail teams with flaky gates, wrong canary metrics, and staging prod parity gaps. Interviewers want you to tell both sides honestly.

Enterprise use cases

Postmortem pipeline action items (model answer):

• P0: Fix Rollouts analysis PromQL; add 5xx rate gate — owner: platform — due: 3 days.
• P0: Contract test required on main fan-in before deploy-staging — owner: checkout squad.
• P1: Staging sync LaunchDarkly flags nightly from prod snapshot — owner: SRE.
• P1: Scan WARN → FAIL on CRITICAL for main branch only — owner: security.
• P2: Monthly game day: inject bad digest in staging, practice rollback under 5 min.

Production case study

Checkout API v2.14.0 — full narrative for interview:

• Root cause: Feature flag off in staging, on in prod — new code path untested.
• Contributing: Contract test file not in paths-filter for lib change triggering API build.
• Detection failure: Canary analysis used avg latency; 4% 5xx on canary subset invisible.
• Mitigation: argo rollouts abort at 4:53; sha256:good88 restored by 4:56.
• Outcome: $14k revenue impact; CFR quarter rose 0.8%; 5 pipeline PRs merged in 10 days.

Trade-offs

• Save: digest rollback — 30 seconds vs 45-minute rebuild-revert cycle.
• Save: deploy markers — SHA identified in 2 min not 2 hours log grep.
• Fail: WARN scan policy — CRITICAL CVE shipped.
• Fail: canary auto-promote without error-rate query — full blast.
• Fix cost: stricter gates add 8 min main pipeline — accepted after $38k/hour loss.

Security implications

Incident digests with CRITICAL CVE (WARN policy) are supply-chain near-miss. Postmortem upgrades scan to fail; Cosign verify prevents unsigned rollback digest injection.

• Rollback digest must come from signed registry — verify admission on rollback path too.
• Emergency hotfix workflow still runs scan — no bypass without CISO ticket.
• Postmortem access: who can workflow_dispatch prod deploy — audit log reviewed.

Scalability analysis

During incident, 12 engineers opened Actions logs simultaneously — GitHub UI throttled. Runbook links direct to specific job ID and digest; status page communication template in pipeline repo.

• Incident Slack bot posts digest + SHA + rollback command automatically on prod deploy.
• Statuspage trigger wired to SLO burn alert — not manual exec comms.
• Postmortem template in pipeline repo — action items become GitHub issues automatically.

Staff engineer insights

• Never blame on-call — blame missing gates; interviewers test systems thinking.
• Quantify incident cost and pipeline fix cost — staff answers include dollars.
• "Pipeline saved us" is rollback story — practice 60-second version.
• Action items must be pipeline-as-code changes — not runbook PDF additions.

Best practices

• Blameless postmortem — fix pipeline not people.
• Every action item is a YAML or policy change with owner.
• Game day quarterly — inject failure, measure rollback time.
• Incident Slack bot links digest, SHA, rollback command on every prod deploy.

Anti-patterns to avoid

• Disable deploys indefinitely instead of fixing analysis query.
• Skip postmortem because "we rolled back fast" — systemic gaps remain.
• Hero on-call SSH fix with no pipeline change — not repeatable.

Common mistakes

• "We added more manual approval" — doesn't fix false-green CI.
• Rollback runbook requires rebuild — loses digest advantage.
• Postmortem without metric fix on canary — repeats next release.

Advanced interview questions

Hands-on exercise

Present Checkout v2.14.0 postmortem in 12 minutes to a peer. Include timeline, pipeline fail/save, 3 action items. Peer asks one curveball: "schema migration was in same release — how does rollback change?"

• Prepare answer: expand/contract migration or rollback incompatible — forward-fix strategy.
• Draw metric graph with deploy marker annotation — practice on paper.

Summary

You can narrate a bad-deploy incident showing how the pipeline failed (canary miss, WARN scan, staging parity) and saved you (digest rollback, deploy markers) — with concrete postmortem action items.