Production MongoDB Systems

Introduction

A production MongoDB system is the day-to-day reality after the architecture diagrams are signed off: deploys, on-call, capacity reviews, incidents, audits and unblocking developers. This lesson zooms in on the operational habits that distinguish good production systems from accidents-in-waiting.

The keystone idea: everything that touches production is reviewed, observable and reversible.

Understanding the topic

Operational habits of healthy production MongoDB:

• GitOps for the data layer — every index, validator, user, alert lives in version control.
• Schema migrations as code — tracked in a migrations collection; runnable forward & back.
• SLO-driven on-call — paged on error budget burn, not raw thresholds.
• Capacity reviews monthly — CPU/RAM/disk + index size + connections trend.
• Index hygiene quarterly — drop unused (via $indexStats), consolidate redundant ones.
• Change windows + canary — deploys never go straight to 100%.
• Postmortems for every page — themes drive next quarter's roadmap.

Syntax reference

Schema migration runner — minimal, idempotent, tracked:

// migrations/2025-01-12-add-status-to-orders.js
export const id = "2025-01-12-add-status-to-orders";
export async function up(db) {
  await db.collection("orders").updateMany(
    { status: { $exists: false } }, { $set: { status: "open" } }
  );
  await db.collection("orders").createIndex({ tenantId: 1, status: 1, createdAt: -1 });
}
// Runner: skip if id present in migrations collection; else apply + record.

Real-world use

The teams whose MongoDB clusters quietly run for years all do the same boring things: GitOps everywhere, SLOs on the wall, monthly capacity reviews, indexed migrations, postmortems with follow-through.

Best practices

• Never run a one-off command in production — write a script, review it, apply via the runner.
• Use db.adminCommand({ killOp }) as the last resort; identify and fix the upstream cause.
• Keep a runbook per top-N alert so any on-call engineer can resolve it.
• Schedule game-days monthly: simulate failover, network blip, slow query storm.

Common mistakes

• "Quick" manual index added in prod — nobody knows it exists six months later.
• Pager fatigue from unfiltered alerts — real incidents get lost in the noise.
• Skipping postmortems on "minor" incidents — they grow into a major one.