Quality Gates

Introduction

Quality Gates are explicit pass/fail thresholds in CI/CD — coverage minimums, lint clean, zero Critical CVEs, performance budgets — that block promotion when unmet. Staff engineers distinguish warn vs fail policies and run a waiver process so gates stay trusted: a gate everyone bypasses is worse than no gate.

The story

A data platform team set 80% global coverage as a hard fail. Legacy modules sat at 40%; every sprint spent weeks on coverage busywork instead of features. Change failure rate didn't improve. They reframed: diff coverage ≥85% on touched lines (fail), global coverage ≥60% (warn), Critical CVEs (fail), High CVEs (warn with 7-day SLA). Waivers required VP sign-off, 30-day expiry, linked JIRA. Delivery resumed; regressions in new code dropped 60% because the gate measured what changed — not decade-old untested utilities.

Understanding the topic

Gate design balances safety and throughput with three knobs: metric, threshold, and enforcement level.

• Coverage thresholds: global floor (warn) + diff coverage on PR (fail) — JaCoCo, Istanbul, Codecov.
• Warn vs fail: fail blocks merge/deploy; warn creates ticket/metric but allows progress — use warn for legacy debt, fail for new code.
• Waiver process: documented exception — owner, reason, compensating control, expiry date, approver role.
• Gate placement: PR gate (fast feedback) vs deploy gate (staging→prod promotion) — different thresholds possible.
• Observability: gate pass/fail metrics over time — if waiver rate >5%, gate is miscalibrated.

Internal architecture

Quality gate flow — multiple signals converge to a single promote/deny decision.

CI signals collected
   ├─ unit tests pass
   ├─ diff coverage ≥ 85%
   ├─ global coverage ≥ 60% (warn)
   ├─ SAST: 0 new ERROR
   ├─ SCA: 0 Critical CVE
   └─ lint clean
   ↓
Gate evaluator (Codecov, SonarQube, custom script)
   ↓
┌ FAIL → block merge / deploy
├ WARN → merge OK, ticket + dashboard
└ WAIVE → allow with audit record + expiry
   ↓
Artifact promote / deploy continues

Visual explanation

Two diagrams show where Quality Gates lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Inventory desired gates: tests, diff coverage, SAST severity, CVE severity, lint, performance.
2. Classify each as fail (new code) or warn (legacy) — document in ADR accessible to all teams.
3. Integrate Codecov or SonarQube for coverage; configure diff coverage check on PR.
4. Implement waiver file (e.g. .quality/waivers.yaml) with schema: id, metric, expiry, approver, ticket.
5. Add CI step parsing waivers — allow fail if valid waiver; deny expired waivers automatically.
6. Dashboard waiver rate and gate failure MTTR — review in monthly platform meeting.

Production implementation

Codecov diff coverage gate + waiver YAML validation in CI:

# codecov.yml
coverage:
  status:
    project:
      default:
        target: 60%
        threshold: 2%  # warn drift
    patch:
      default:
        target: 85%    # fail on PR diff
        threshold: 0%

# .quality/waivers.yaml
waivers:
  - id: WVR-2024-0412
    gate: sca-critical
    reason: Awaiting upstream fix for transitive lib
    ticket: SEC-8821
    approver: security-lead@corp.com
    expires: 2024-07-01

# CI validation
  - run: |
      python scripts/check_waivers.py --gates coverage-patch,sca-critical
      # exits 1 if gate failed and no valid waiver

Execution workflow

Real-world use

SonarQube quality gates are enterprise standard for Java shops. Codecov/Coveralls dominate diff coverage on GitHub. Google's internal Critique uses custom thresholds per project. DORA research: elite performers use automated deployment gates — not manual checklists. Netflix Spinnaker supports automated canary analysis gates — same philosophy, runtime metrics instead of static coverage.

Enterprise use cases

Release train (fortnightly): all services must pass org quality gate to board the train. Gate includes: diff coverage 85%, zero Critical/High CVE, SAST no new blocker, contract tests green. Warn: global coverage below 70%, code smells above threshold. Train conductor (release manager) views waiver registry — max 3 waivers per train or slip release.

• Monorepo: per-package gates via Nx tags — critical packages stricter thresholds.
• Mobile: performance budget gate (bundle size) fails PR if +5% over baseline.
• Compliance: gate results exported to GRC; waivers require compliance delegate for SOX systems.

Production case study

Payment processor (PCI): QSA required demonstrable quality controls before prod deploy.

• Challenge: manual QA sign-off checklist; no automated thresholds; audit finding on traceability.
• Decision: SonarQube quality gate + Codecov patch 85% + zero Critical CVE fail; waiver board weekly.
• Warn policy: global coverage below 75% opens tech-debt ticket, doesn't block.
• Outcome: PCI audit closed finding; release lead time reduced 2 days by removing manual gate.

Trade-offs

• Strict global coverage fail: punishes legacy; encourages meaningless tests.
• Diff coverage only: legacy holes remain; acceptable if paired with ratcheting global warn.
• Too many fail gates: parallel flakes cause merge lottery — consolidate or serialize wisely.
• Warn without ticket automation: warnings ignored — integrate JIRA on warn.
• Permanent waivers: gate erosion — expiry mandatory.

Security implications

Quality gates are compliance controls — treat waivers as security artifacts:

• CVE waivers need compensating controls documented — WAF rule, network isolation, not just "accept risk."
• SAST waivers expose org to exploit until expiry — track in vulnerability register.
• Gate bypass via admin merge must log to immutable audit (GitHub audit log, SIEM).
• Coverage gates don't prove security — don't waive SAST for coverage trade-off without analysis.

Scalability analysis

Gate management at org scale:

• Central waiver registry API — not 500 repos each with local waivers.yaml drift.
• Gate results as OPA input for deploy — single policy engine across CI and CD.
• Per-team gate dashboards — platform team sees systemic issues vs one bad repo.
• Codecov/Sonar costs scale with LOC — negotiate org license vs per-repo.

Staff engineer insights

• Diff coverage is the highest ROI gate for legacy codebases — measures what you ship, not what ancestors didn't test.
• If waiver rate exceeds 5% of merges, the gate threshold is wrong or the team needs help — don't blame developers.
• Warn gates without automated ticket creation are vanity — integrate with JIRA/Linear on warn.
• Deploy-time gates (staging→prod) can be stricter than PR gates — catch integration issues unit tests miss.

Best practices

• Use diff/patch coverage for PR fail; global coverage as warn with ratchet.
• Every waiver has expiry ≤90 days and linked ticket.
• Same gate config in CI and CD promote — no drift between check and deploy.
• Publish gate policy doc before enabling fail — reduces tribal knowledge.
• Review gate failures weekly — recurring failures indicate training or tooling gaps.

Common mistakes

• 80% global coverage fail on day one — gate disabled or gamed with empty tests.
• Warn-only CVE policy — Critical CVEs live in prod indefinitely.
• Waivers without expiry — permanent exceptions become policy.
• 10 independent fail gates serially — 45-minute merge lottery; consolidate reporting.

Advanced interview questions

Hands-on exercise

Lab: Configure Codecov patch coverage 85% fail on a sample repo. Submit PR that lowers diff coverage — confirm block. Add valid waiver YAML; confirm merge allowed. Expire waiver; confirm block returns.

• Document fail vs warn policy table for 5 metrics.
• Create waiver template with compensating control field.
• Calculate waiver rate formula for dashboard.

Summary

You can design quality gates with intentional warn vs fail policies, diff coverage for legacy codebases, and waiver processes that expire. Explain gates as trust instruments — miscalibrated thresholds erode them faster than having no gate at all.