Production Reliability

Introduction

Production reliability is the discipline of running MongoDB so it stays up — and stays fast — through deployments, outages, traffic spikes, and human mistakes. Reliable systems are boring: they alert before failing, fail over without paging anyone, and recover without data loss.

Reliability is built from four pillars: topology (replica sets across zones), observability (metrics, logs, traces), process (runbooks, drills, change management), and capacity (headroom + autoscaling).

Understanding the topic

The reliability pillars:

• Topology — 3-node minimum, multi-AZ, optional multi-region for tier-1 systems.
• Observability — Atlas metrics + Datadog/Prometheus + slow-query log → dashboards & SLOs.
• Alerting — page on saturation (CPU > 80%, oplog window < 1h, replication lag > 10s), not on every blip.
• Change management — every index, schema or version change behind a feature flag + canary.
• Capacity — keep 30%+ CPU and disk headroom; enable Atlas auto-scaling for storage.
• Chaos drills — kill nodes, fail over primary, simulate region loss — quarterly.

Syntax reference

Define SLOs that drive engineering priorities:

SLI                          Target          Error budget / 30d
─────────────────────────────────────────────────────────────────
Read availability             99.99 %         4 m 19 s
Write availability            99.95 %         21 m 36 s
p99 read latency (catalog)    < 50 ms         5 % of reads
p99 write latency (orders)    < 120 ms        5 % of writes
Backup RPO                    ≤ 60 s          —
Failover RTO                  ≤ 30 s          —

Real-world use

Netflix-style postmortem culture combined with Atlas's built-in HA gives mature teams >99.99% availability without on-call heroics. Stripe-grade fintechs run game-days monthly — fail over a primary on purpose, watch the SLO burn rate, fix any surprises.

Best practices

• Define SLOs before the cluster goes live; alert on burn rate, not raw thresholds.
• Practice blue/green deployments for major version upgrades.
• Tag resources with tier and owner — every alert routes to the right team.
• Postmortem every incident, however small — track action items to completion.

Common mistakes

• Single-AZ deployments — one bad rack reboot ends your year.
• Alerting on raw CPU spikes — pages on noise, ignores real degradation.
• Letting capacity drift to 95% before scaling — there's no headroom for traffic surprises.