Build Failure Triage

Introduction

Build failure triage is the operational discipline of assigning ownership, quarantining flaky tests, and bisecting regressions — so red CI means actionable signal, not background noise engineers ignore.

The story

CI was red 40% of the time; teams clicked "Re-run failed jobs" until green. A real regression — bad SQL migration — hid among flakes for three days. Platform instituted triage: CODEOWNERS auto-assign on failure, flaky test quarantine file with 7-day SLA, and git bisect runbook tied to merge-base. MTTR for real failures dropped from 72h to 4h; flake rate tracked weekly dropped from 12% to 2% after quarantine enforcement.

Understanding the topic

Triage turns CI from alarm fatigue into routing + root cause — every red build gets an owner, classification (flake vs real), and resolution path.

• Ownership: CODEOWNERS + failed job path → Slack/PagerDuty route; no "shared CI" orphan failures.
• Flake quarantine: mark test @flaky or move to quarantine suite; must not block merge; tracked with expiry and fix SLA.
• Git bisect: binary search between last green and red SHA to find introducing commit — automate with git bisect run + test script.
• Classification: infra (runner OOM), dependency (registry down), test flake, product regression — different runbooks.
• Metrics: flake rate, time-to-triage, re-run count per PR — DORA change failure rate input.

Internal architecture

Failure triage flow from red build to resolved root cause:

CI job fails
   ↓
Auto-notify CODEOWNERS (path from failed job/working dir)
   ↓
Classifier bot / on-call
   ├─ Re-run once → passes? → open flake ticket → quarantine
   ├─ Infra error (OOM, timeout) → platform ticket
   └─ Consistent fail → bisect
         git bisect start
         git bisect bad HEAD
         git bisect good <last-green-sha>
         git bisect run ./scripts/ci-check.sh
   ↓
Fix commit or quarantine with owner + expiry
   ↓
Post-mortem if prod-impacting; update flake registry

Visual explanation

Two diagrams show where Build Failure Triage lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Enable CODEOWNERS on repo — map directories to teams; require review on failure paths.
2. Add CI failure Slack notification with workflow run URL, failed job, and git SHA.
3. Create quarantine mechanism: Jest @flaky tag, pytest mark, or separate non-required workflow for known flakes.
4. Document bisect runbook: git bisect run script wrapping minimal repro command.
5. Dashboard: flake rate = flaky re-runs / total runs; target <3%; weekly review of quarantine list.

Production implementation

Automated bisect in CI on main breakage + flake quarantine in Jest:

• GitHub Actions: failed job sends to Slack via workflow run webhook + ownership from paths-filter.
• Buildkite/analytics plugins track flaky test history automatically.
• Require flake fix ticket link when adding to quarantine — bot enforces in PR.

# scripts/ci-check.sh — exit 0 = good, 1 = bad (for bisect)
#!/bin/bash
set -e
npm ci --prefer-offline
npm test -- --testPathPattern="integration/auth"

# On main red, run locally or in workflow_dispatch:
git bisect start
git bisect bad origin/main
git bisect good v2.3.0   # last known green tag
git bisect run ./scripts/ci-check.sh
# git bisect reset when done

# jest.config.js — quarantined tests (non-default project)
module.exports = {
  projects: [
    { displayName: 'required', testPathIgnorePatterns: ['/quarantine/'] },
    { displayName: 'quarantine', testMatch: ['**/quarantine/**/*.test.ts'],
      runner: 'jest-runner' /* runs but not in required CI job */ },
  ],
};

# .github/CODEOWNERS
/apps/api/          @team-payments
/libs/auth/         @team-identity
/.github/workflows/ @team-platform

Execution workflow

Real-world use

Martin Fowler's "Eradicating Non-Determinism in Tests" and Google Testing Blog on flaky tests established quarantine patterns. git bisect is built into git since 2005 — underused because teams lack a one-command repro script. GitHub's merge queue increased triage urgency because main red blocks everyone.

Enterprise use cases

Google TAP tracks flake rate per test; quarantined tests don't block submit. Spotify's CI assigns ownership via backstage catalog entity. Netflix ChaosMonkey-adjacent CI practice: re-run failed tests 3x before declaring flake — reduces false quarantine.

• Merge queue: failure blocks queue — triage SLA stricter (30 min) than optional PR CI.
• Monorepo: failed Nx project maps to team via project.json tags → automatic routing.
• Compliance: audit log of who quarantined test and when — flakes aren't silent quality debt.

Production case study

SaaS company (60 engineers) reduced ignored-red-build culture. Quarantine file grew to 89 tests without SLA — platform reset: 7-day expiry, auto-fail PR adding quarantine without ticket, bisect runbook in onboarding.

• Problem: 38% re-run rate; engineers merged despite red optional checks.
• Ownership: CODEOWNERS + #ci-failures Slack with 30 min ack SLA for main.
• Flakes: quarantine/ dir; required CI excludes it; weekly flake burndown meeting.
• Bisect: scripts/ci-check.sh used in 4 incidents; avg 12 commits narrowed in 20 min.

Trade-offs

• Quarantine: unblocks velocity; risk if quarantine becomes permanent parking lot.
• Zero tolerance flakes: high quality bar; blocks merges during infra instability.
• Auto re-run (3x): reduces false positives; hides intermittent product bugs if always passes on retry.
• Manual triage: flexible; doesn't scale past ~20 PRs/day without dedicated platform on-call.

Security implications

Triage shortcuts can bypass security — quarantining SAST or signing steps is never acceptable.

• Quarantine allowlist excludes security scan, license check, and secret scan jobs — platform-enforced.
• Bisect scripts must not pull unverified dependencies — use lockfile-pinned npm ci inside bisect run.
• Failed security job = P1 route to security team, not default dev owner quarantine path.
• Audit quarantine PRs — malicious actor could quarantine tests covering their backdoor.

Scalability analysis

At high PR volume, failure notification noise and quarantine debt compound without automation.

• Central flake registry across repos — dedupe same upstream flake in 50 services.
• ML-based flake detection (Buildkite, Launchable) required above 500 runs/day manual triage.
• Bisect on 10K commit range expensive — narrow with affected path log first.
• Cross-timezone ownership — follow-the-sun unless merge queue forces single-region SLA.

Staff engineer insights

• If re-run without code change fixes CI, it's a flake — track it before it trains ignore-red behavior.
• Quarantine without expiry is technical debt with interest — auto-expire or auto-escalate.
• Bisect only works with fast repro script — invest in scripts/ci-check.sh before incidents.
• Platform owns infra failures; product owns test failures — classify in first 5 minutes.

Best practices

• One-command minimal repro script committed in repo for bisect.
• Quarantine list in git with owner, date, ticket link, expiry.
• Never quarantine security or migration test jobs.
• Track flake rate weekly; exec dashboard if rate >5%.

Common mistakes

• Re-run until green without filing flake — teaches org to distrust CI.
• Quarantine entire job instead of single test — hides real failures.
• Bisect without fetch-depth or on squashed history only — wrong result.

Advanced interview questions

Hands-on exercise

Introduce a deliberate test failure, practice bisect to find it, and draft a quarantine entry with SLA.

• Time bisect vs linear scan — document crossover point.
• Write scripts/ci-check.sh usable in bisect run.
• Simulate flake: random fail test — re-run 3x, classify.

# Seed repo with 5 commits, one breaks test
git bisect start
git bisect bad HEAD
git bisect good HEAD~4
git bisect run npm test -- --testPathPattern=unit/smoke

# After finding commit, revert or fix
git bisect reset

# Quarantine template (quarantine/README.md):
# | Test | Owner | Ticket | Added | Expires |
# | auth flake | @team-id | JIRA-123 | 2024-03-01 | 2024-03-08 |

Summary

You can triage build failures through ownership routing, flake quarantine with SLA, and git bisect automation — keeping CI trustworthy so red means stop and fix, not re-run until lucky.