Replication Challenges

Introduction

Replication challenges are the rehearsals for real failover events. Forcing controlled failovers in staging — and watching the metrics, the client behaviour, the lag — is the only way to know your system will actually survive an unplanned one.

Understanding the topic

5 replication drills:

• 1. Forced election with rs.stepDown(60) — measure RTO.
• 2. Kill the primary process — observe client reconnect and retry behaviour.
• 3. Pause a secondary for 30 min — verify oplog window holds, then catches up.
• 4. Add a hidden analytics node tagged for BI reads.
• 5. Convert a 3-node primary-secondary-arbiter to 3 data-bearing nodes.

Informative example

Drill 1 — measured failover with stepDown:

// On the current primary
rs.stepDown(60);   // step down for 60s
// → election fires, new primary in ~10s

// From the app, measure:
const t0 = Date.now();
let writes = 0;
while (Date.now() - t0 < 60_000) {
  try {
    await db.coll.insertOne({ ping: new Date() });
    writes++;
  } catch (e) {
    // record retryable write retries; driver should auto-retry
  }
}
console.log("writes during failover", writes);

Best practices

• Drill quarterly; record RTO and any client-visible errors.
• Always enable retryable writes in the driver (retryWrites=true).
• Monitor rs.printSecondaryReplicationInfo() during drills.