Recovery Techniques

Introduction

Recovery is the act of bringing a database back to life after a failure: corrupted node, accidental drop, ransomware, region outage. The technique depends on the blast radius — restoring a single collection takes minutes, rebuilding a sharded cluster across regions can take hours.

The most important recovery skill is practice. Teams that drill recovery quarterly recover in minutes; teams that have never tried take days and often lose data.

Understanding the topic

The recovery decision tree:

• Single document/collection deleted → restore the collection from a recent snapshot into a temp DB, then $merge back.
• Whole database dropped → Atlas PIT restore to "1 minute before the drop" — fastest.
• Node disk corruption → remove member from replica set, wipe data dir, re-add — initial sync from peers.
• Full cluster loss → restore latest snapshot into a new cluster, replay oplog up to incident time.
• Region outage → fail over to multi-region replica (Atlas Global Cluster) — minutes, not hours.
• Ransomware / data tampering → restore to last known clean PIT, audit oplog for the attack window.

Informative example

Restoring a single dropped collection without touching production:

# 1. Restore the snapshot into a temporary cluster or DB
mongorestore --uri="$STAGING_URI" --gzip --archive=dump-20251215.gz   --nsInclude="shop.orders" --nsFrom="shop.orders" --nsTo="staging.orders_restore"

# 2. Diff vs. current to find what's missing
mongosh "$PROD_URI" --eval '
  const missing = db.orders.aggregate([
    { $lookup: { from: "orders_restore", localField: "_id", foreignField: "_id", as: "r" } },
    { $match: { r: { $size: 0 } } }
  ]);
'

# 3. Surgical $merge back into prod
db.orders_restore.aggregate([
  { $merge: { into: { db: "shop", coll: "orders" }, whenMatched: "keepExisting", whenNotMatched: "insert" } }
]);

Real-world use

A well-known fintech recovered from a junior engineer's db.dropDatabase() in 11 minutes using Atlas PIT restore — zero customer-visible impact. A non-Atlas team with the same incident lost 6 hours of data because their backups were 24h stale.

Best practices

• Drill quarterly. Pick a random Friday, restore a recent snapshot to a sandbox, time it.
• Keep runbooks in version control — every recovery step copy-pasteable.
• Never recover into production directly — always restore to a sandbox, validate, then promote.
• Use oplog replay after a snapshot restore to recover the last few minutes.

Common mistakes

• Recovering without first capturing the corrupted state — you may need the evidence later.
• Skipping the validation step — restoring bad data on top of bad data is worse than nothing.
• Forgetting to re-enable application writes only after monitoring confirms health.