Disaster Recovery and Business Continuity

Advanced System Design & Software Architecture · Lesson 64 · Architecture Practice, Interviews and Capstone

Lesson Overview

TechLearningPro's order database had a streaming replica in a second availability zone, a nightly snapshot, and a dashboard that said replication lag was near zero. The team called this "the DR plan." On a Thursday the primary's volume was corrupted by a bad migration that deleted a column the application still wrote. The replica applied the same WAL and dropped the same column. The snapshot was restorable and lived in the same region as the broken volume. The restore, which nobody had timed, took eleven hours. Checkout was the business for those eleven hours. Support had no script, finance had no way to capture a manual order, and the status page still said operational because the health check read the replica, which was up and wrong.

Replication is not a backup. A backup in the same failure domain is not a recovery. A recovery nobody has timed is not an RTO. This lesson separates business continuity — how the company keeps selling, teaching, and telling the truth while systems are hurt — from disaster recovery — how specific systems return inside a recovery time objective and a recovery point objective. The numbers are business decisions. The design either fits inside them or the numbers have to change in public.

Database Replication and Consistency already taught WAL, synchronous commit, promotion, and fencing. CAP Theorem and Consistency Models explains why an active-active ledger is a conflict-resolution problem, not a checkbox. Non-Functional Requirements and SLO Design set the minutes you are allowed to be down or wrong in ordinary faults. This lesson is the wider fault: a zone gone, a region gone, a mistaken delete, a ransom of the account that owns the disks. You will write a DR plan for TechLearningPro's order platform that a game day can fail.

Learning Objectives

By the end of this lesson you will be able to:

1. Define RPO and RTO for a named service, and show why different parts of one product do not share a number.
2. Separate business continuity from disaster recovery, including communication, manual procedures, and the technical restore.
3. Explain why replication copies mistakes and why an immutable backup with a timed restore is a different control.
4. Choose among multi-AZ failover, warm passive region, and active-active, and say which consistency and cost you bought.
5. Place backups outside the failure domain they are meant to survive, and include restore duration inside the RTO.
6. Write a failover runbook that fences the old primary, promotes one writer, and has a failback path.
7. Design a chaos experiment or game day around a steady state taken from an SLO, with a blast radius you can stop.
8. State the security and privacy constraints a restore must not violate: secrets, tenant isolation, and deletion retention.

Prerequisites

• Database Replication and Consistency: promotion, synchronous_commit, fencing, split brain. This lesson uses that mechanism. It does not re-derive WAL.
• CAP Theorem and Consistency Models and Distributed Transactions and Saga Pattern: what spans regions will not be a single ACID transaction.
• Non-Functional Requirements and SLO Design: ordinary-fault budgets. RTO has to be compared to those budgets honestly.
• Load Balancing and Traffic Management and Microservices Deployment and Release Strategies: how traffic moves, and why a deploy and a failover are different operations.
• Idempotency and Exactly-Once Processing: retries during failover will replay captures.
• Architecture Security and Threat Modeling: backups are copies of personal data and must not become a new trust boundary.
• Design a Large-Scale E-Commerce Platform and Design a Payment Processing System: the order and money flows this plan protects.
• Comfort with the idea of a backup window. You will not operate a cloud vendor's proprietary control plane in this lesson. PostgreSQL's PITR vocabulary is used because it is concrete and portable.

Detailed Core Concepts

RPO and RTO are business limits, not cluster settings

Recovery point objective is the maximum age of data you are willing to lose, measured backward from the moment of the disaster. An RPO of five minutes means you have accepted that commits acknowledged in the last five minutes may be gone, and you have designed a way to discover which ones. An RPO of zero for acknowledged captures means a commit is not acknowledged until it sits in a place the disaster you have in mind will not erase.

Recovery time objective is the maximum time from the decision to recover until the service meets its definition of restored. The clock does not start when an engineer opens a laptop. It includes detection, the human or automatic decision, the restore or promotion, the application pointing at the new writer, cache warm-up, and a check that the answer is correct. If detection takes twenty minutes and restore takes forty, you do not have a thirty-minute RTO.

Both numbers are chosen per service with someone who can accept the business harm. Checkout capture, catalog browsing, and the analytics export do not share them.

If product cannot accept the catalog row, do not let them inherit the ledger's RPO by default. Inheriting the strictest number for every store is how DR budgets explode and then get ignored. Write the table, and let the strict row justify the expensive control.

Compare these numbers to lesson 62's error budgets. A 99.9 percent time-based monthly budget is 43.2 minutes. A regional failover with a 30-minute RTO consumes most of that budget in one event. That can be legitimate if region loss is rare and you say so. It is not legitimate to promise 99.99 percent (4.3 minutes) and also a human-driven regional failover. Either the SLO describes a narrower fault model — one instance, one zone, with automatic healing — or the SLO changes. Put both documents in the same review.

Business continuity is the company, not the cluster

Disaster recovery restores systems and data. Business continuity keeps the organization's obligations moving when the systems are not restored yet: learners, instructors, support, finance, regulators, status communication.

For an order platform the continuity pack is short and specific:

• Who may declare a disaster, and who is the backup decider if that person is offline.
• What support tells a learner who paid and cannot see the course. The sentence has to match the actual restore state, or you create the lesson 61 failure where the macro lies.
• A manual capture path with a ceiling: finance can record a named-learner purchase in a spreadsheet template that includes an idempotency key, and engineering replays it after restore. The ceiling — perhaps fifty orders — is the point at which you stop selling rather than create an unreconciled pile.
• How instructors are told that live cohorts continue even if new purchases wait.
• When a status page moves from "degraded" to "down," and who is allowed to move it. The health check must not be the only voter if the health check can be green while the data is wrong.
• Which regulators or enterprise customers require a notice, and the template for that notice. The template is legal's. The trigger time is yours.

Continuity procedures are tested with the same seriousness as failovers. A spreadsheet nobody can find is not a manual path. A status page only three people can edit, all in one city, is a correlated failure with the office network.

Backups, replication, and the failures each one survives

Replication copies the primary's changes to another node so you can lose the node. It is the right control for a dead disk or a dead zone, within the RPO implied by synchronous or asynchronous commit. Lesson 29's vocabulary stays authoritative: asynchronous replicas can lose acknowledged commits if the primary dies before WAL ships; synchronous commit with a named standby avoids that loss and can stall writes when the standby is unreachable.

Replication is the wrong sole control for a bad UPDATE, a dropped column, a compromised account, or ransomware that encrypts every replica the credentials can reach. Those faults are copied faithfully. The control is a backup retained far enough back to predate the mistake, stored where the same credentials and the same region failure cannot destroy it, and proven by a restore.

For PostgreSQL the portable shape is:

• A base backup taken with pg_basebackup or a storage snapshot that the database has been told is consistent.
• Continuous archiving of WAL so you can do point-in-time recovery to a recovery_target_time before the bad migration.
• A retention longer than the time it takes a human to notice a quiet corruption. A seven-day backup does not help an error discovered on day eight. Choose retention from detection, not from disk comfort.
• A copy of that archive in another region, with object-lock or an equivalent immutability window, so a stolen write credential cannot delete history during the lock.

Logical dumps via pg_dump are a useful second form for a single schema or a tenant export. They are a poor sole strategy for a large ledger, because restore time and consistency across tables are worse than PITR. Use them for the "extract one tenant" case from the privacy side, not as the RTO plan for checkout.

Restore duration is measured, not estimated from the snapshot size on a slide. A 2 TB volume that restores at a provider's best-effort rate can miss a 30-minute RTO even when the bytes exist. If the measured restore is two hours, the RTO is at least two hours plus detection and promotion of traffic, or you invest in a warm standby that is already applying WAL and only needs promotion. Those are different costs. The plan names which one you pay.

Test restores on a calendar. A quarterly restore of the order ledger into an isolated network, followed by a query that checks a known purchase, is the minimum this course will call a backup. An untested snapshot is a hypothesis with a storage bill.

Failure domains: instance, zone, region, account

Name the domain or the plan will use the word "redundant" for all of them.

• Instance. A process or a VM. Multiple replicas behind a load balancer, one writer. RTO can be seconds if the orchestrator restarts pods. RPO depends on the database, not on the pod.
• Zone. A data center inside a region. Multi-AZ primary and synchronous standby, automatic or scripted promotion with fencing. This is the default for the order ledger's "zero RPO inside a zone loss."
• Region. A geography. Zone-redundant storage still fails together if the region does. Regional DR needs a copy of the data that is not in that region, and a way to run the application there: images, configuration, secrets, DNS, and a tested cutover.
• Account or control plane. A compromised cloud account deletes replicas and backups together if they share an identity and an unlock policy. Immutability and a separate backup account are the controls. This is the scenario replication cannot see.

Multi-AZ is not multi-region. A slide that draws two clouds and a single bucket in one region has drawn a picture of a region failure. Design a Global Streaming Platform may justify active multi-region reads for media. Orders do not inherit that topology because video needed it.

Active-passive, warm standby, active-active

Active-passive, backup only. One region writes. The other region has binaries and a cold copy of backups. RTO includes a full restore. Cheapest. Fits analytics. Does not fit a 30-minute order RTO if restore takes two hours.

Warm passive. The second region applies WAL continuously, takes no user traffic, and can be promoted. RPO is the replication lag. RTO is detection plus promotion plus DNS or connection switch plus application warm-up, if fencing is already scripted. This is the honest design for TechLearningPro orders across regions. You pay for a database that is almost a primary, and you refuse to serve writes from both.

Active-active writes. Both regions accept orders. You now own conflict resolution: two captures, two seats, a split brain during a partition. Lesson 31 is the theory. For a ledger, the practical versions are narrower than "both sides write everything." You can partition by tenant home region, as Design a Multi-Tenant SaaS Platform describes, so each order has one writer and the other region only reads a replica. That is active-active serving with single-writer partitions, which is a sound design and is not the same as merging two primaries. Or you can keep writes in one region and accept the latency for distant learners. The plan should prefer one of those over ad hoc last-write-wins on a money row.

Failover is not complete when the database promotes. Application configuration, background workers, and the outbox dispatcher must agree on the new writer. Two workers, one in each region, both sending receipts, are a duplicate-mail incident on top of a disaster. The runbook names the single writer for each workload.

Failback is a second migration, often more dangerous than the failover, because the original region returns with stale disks that must not become primary. You rebuild them as replicas of the survivor, you do not "switch back" to an old primary that has diverged. The plan includes failback or you will improvise it at 2 a.m. after the region recovers.

Fencing, because two writers are worse than downtime

Lesson 29's rule stands: promote only after the old primary is fenced. Fencing means the old node cannot accept writes. STONITH, a cloud API that stops the instance, a storage revoke, a database role you can disable. "We will be careful" is not fencing. A network partition that hides the old primary from you and not from clients is the split-brain case. Clients must not keep a long-lived connection pool aimed at a DNS name that still resolves to the corpse. Short DNS TTLs help and are not sufficient; the application's configured endpoints and any in-memory leader must refresh.

During the uncertain window, the availability choice is to stop writes. Serving a second capture is outside the RPO you published. Queueing captures with idempotency keys, if the queue itself is in the surviving region, can protect the user from retyping a card. The queue is then part of the RPO math: if it is in the dead region, it does not exist.

Chaos and game days

Chaos engineering, in the sense of the Principles of Chaos community, is an experiment: define a steady state, hypothesize that it holds under a fault, inject the fault, watch, and limit the blast radius. The steady state for this course is an SLO from lesson 62, not "CPU looked normal."

A game day is a scheduled experiment with humans and a runbook. A chaos experiment may be automated and smaller. Both are allowed to find that the runbook is wrong. That is the success of the exercise, if you stop before you harm learners.

Examples that fit an order platform, from safest to sharpest:

• Restore last night's backup into an isolated environment and time it. No production impact. Do this until the time is boring.
• In staging that mirrors topology, kill the primary and follow the runbook. Measure detection, fencing, promotion, and first successful read.
• In staging, run the bad migration against a copy and practice PITR to one minute before it.
• In production, with the business's consent and a small blast radius, stop one application instance or one zone's nodes if the orchestrator claims it can reschedule. Do not start here. Do not inject data-destroying faults in production.
• Simulate the acquirer timeout and confirm the UNKNOWN path from lesson 62 still reconciles after a database failover. Payment's idempotency has to survive the promotion.

Abort criteria are written before the start: error budget burn above a line, any sign of a second writer, any uncertainty about fencing. The experiment owner can halt without a committee. A game day without an abort is a planned incident.

Runbooks

A runbook is the thing a tired person who did not design the system can execute. For regional failover of orders it contains:

• The trigger: which signal, lasting how long, authorizes the move. Lag, error rate, and an explicit human confirm for region evacuation.
• The owner and the backup owner.
• Fencing steps and the proof that fencing worked: the old primary cannot be reached, or its role is confirmed stopped.
• Promotion steps and the proof of one writer: a query for the primary role, a token or LSN the application will use.
• Traffic shift: DNS, load balancer, or configuration, and the TTL you actually set last quarter.
• Workers: which region runs the outbox, and the command that scales the other to zero.
• The user-facing sentence and the status-page text.
• How to handle in-flight captures: consult the ledger and the acquirer, do not replay blindly.
• Failback outline, even if the detail is "rebuild replica, do not promote the old disk."
• Evidence to keep: timestamps, LSNs, who decided. You will need them for the customer letter and for the next ADR.

If the runbook says "contact the database vendor" with no account number and no contract tier, you have a hope. Write the concrete dependency.

Security and privacy during recovery

Backups are a trust boundary. A snapshot of the enrollment database contains the personal data lesson 63 tried not to scatter. The backup account is not the application account. Restore environments are not world-readable. A support engineer restoring "a copy to look at one learner" is an access path that needs the same authorization as production, or a scoped logical export.

Deletion has a deadline. If the privacy residual says backups age out in 30 days, the DR retention and the privacy notice must match. Restoring a 30-day-old snapshot into production will resurrect deleted accounts unless a deletion log is replayed on top. That replay is part of PITR design, not a follow-up ticket.

Secrets in backups are a second problem. Prefer the database to hold data encrypted with a key that lives in a KMS the backup account cannot use alone, or accept that the backup store is as sensitive as the primary and control it that way. Do not "fix" this by checking keys into the runbook repository.

Tenant isolation must hold in the restored database. A restore that disables row security for speed and forgets to re-enable it is an elevation incident caused by the recovery. The validation query after promote includes a cross-tenant deny check, not only "the orders table has rows."

Architecture and Design Diagrams

Failure domains and the control that actually survives each one:

mermaidCopy
flowchart TD
    inst[Instance loss] --> pod[Orchestrator restarts the app]
    zone[Zone loss] --> sync[Multi-AZ sync standby plus fencing]
    region[Region loss] --> warm[Warm replica in region B plus DNS cutover]
    logic[Bad migration or delete] --> pitr[PITR from WAL archive to a time before the fault]
    cred[Compromised admin credentials] --> imm[Immutable backup in a separate account]
    pod --> rpo0[RPO near 0 if the database quorum survived]
    sync --> rpo0
    warm --> rpolag[RPO equals replication lag]
    pitr --> rpotime[RPO equals the chosen recovery target]
    imm --> rpotime

Order platform topology this lesson accepts:

mermaidCopy
flowchart LR
    subgraph regA [Region A active]
      lbA[Load balancer]
      appA[Order and payment apps]
      pri[(Primary ledger)]
      lbA --> appA --> pri
    end
    subgraph regB [Region B warm]
      lbB[Load balancer standby]
      appB[Apps scaled to read-only or zero workers]
      stb[(WAL replica)]
      lbB --> appB
      appB -.-> stb
    end
    pri -->|sync within region, async across regions| stb
    pri --> arc[WAL archive]
    arc --> lock[Immutable copy in backup account]
    users[Learners] --> lbA

Failover sequence, including the fence:

mermaidCopy
sequenceDiagram
    actor Decider
    participant Old as Region A primary
    participant New as Region B replica
    participant Dns as Traffic control
    participant App as Order apps
    Decider->>Old: fence writes and prove stopped
    Decider->>New: promote, confirm single primary
    Decider->>App: point writer at B, scale A workers to zero
    Decider->>Dns: shift traffic, TTL already low
    App->>New: read a known order, deny cross-tenant probe
    Note over Decider,App: In-flight captures reconciled with acquirer before replay

Technical Explanation

Turning RPO into a replication setting

Inside one region the order ledger wants RPO zero for acknowledged commits against a zone loss. That is synchronous commit to a standby in another zone, with the meanings lesson 29 gave synchronous_commit and synchronous_standby_names. The application does not acknowledge a capture to the learner until that commit returns. The cost is write latency and the risk that a zone loss of the standby stalls commits if you required that standby and have no alternate. The configuration should name a quorum that can be satisfied with one of two standbys, or you have traded a zone loss of the primary for a zone loss of the only synchronous target. Write that choice down. It is the actual RPO mechanism.

Across regions, synchronous commit usually costs more latency than checkout will accept, and a region partition would stall the business. Asynchronous replication to region B means the RPO is the observed lag, and the runbook's first query is that lag. If lag is twenty minutes and the RPO you published is one minute, you do not fail over and pretend. You either wait, accept a new RPO in the incident channel, or recover from the WAL archive if it is ahead of the replica. The archive can be more recent than a stalled replica. Measure both.

Point-in-time recovery, in steps a plan can list

1. Isolate a new instance. Do not restore on top of the broken primary.
2. Restore the latest base backup taken before the fault.
3. Replay WAL until recovery_target_time set to a minute before the bad migration, using PostgreSQL's recovery target settings. Inclusive versus exclusive targets matter; pick the one that stops before the destructive transaction and document the choice with a transaction timestamp you recorded.
4. Prevent promotion until a validation query passes: a known order exists, a canary row written after the target does not, the migration's dropped column is still present.
5. Fence the bad primary.
6. Promote the restored instance only after that proof, then follow the traffic steps.
7. Replay the deletion log if the target time would resurrect accounts that were deleted on purpose before the fault. If the bad migration and a legitimate deletion are interleaved, get a human. This is a place not to be clever.

The plan names where the base backup and WAL live, the expected restore rate from the last timed test, and the person who can read the timeline.

What happens to in-flight payments

A capture has three systems: the learner, your ledger, and the acquirer. Failover can interrupt any message between them. The safe rule is the one lesson 37 and the payment case study already use. If the learner received a terminal CAPTURED or DECLINED and the ledger has that row, you are done. If the ledger has UNKNOWN or the response never returned, you ask the acquirer with the same idempotency key. You do not create a new charge because the first request timed out during DNS cutover. The runbook's payment section is that sentence plus the acquirer console URL and the key prefix. A well-meaning operator who "reprocesses the last hour" without keys will double-charge people during a disaster, which is worse than the outage.

Entitlements follow the ledger, not the other way around. Access is granted from a committed CAPTURED row. A restored database that lost the last thirty seconds of WAL may have lost a capture the acquirer completed. Reconciliation compares acquirer settlements to the ledger and inserts the missing terminal rows. That job is part of RPO zero from the user's point of view even when the database RPO was thirty seconds. Say this explicitly or finance and engineering will argue about two different zeros.

Detection

An RTO clock needs a start. Detection of a dead primary can be a failed health check and a failover tool. Detection of a wrong primary — the migration that deleted a column — is a functional check: a synthetic purchase in a test tenant every minute, and a schema assertion on boot. The health endpoint that selects one row from a replica will stay green. Lesson 62's capture SLI, if it counts reconciled outcomes, will go bad when entitlements stop matching. Wire that SLI to the decider. Do not wire only CPU.

For region loss, detection may be your cloud provider's status and your own failed synthetic from outside the region. Agree in advance that the external synthetic plus provider confirmation is enough, so the meeting during the event is not "is it really down?"

Cost and the decision record

Warm standby is roughly a second database bill plus cross-region traffic for WAL. Immutable backups are storage plus the occasional restore test. Active-active with tenant partitioning is an application program, not a storage feature. The ADR, using lesson 10's shape and lesson 61's frame, records which option won and the RTO that would force a revisit. "We will revisit if a customer contract requires a 15-minute regional RTO and the last game day measured 45" is a reopen condition with a number.

Chaos, scoped so it stays a test

The experiment statement: "If the primary database instance in zone A is stopped, the warm standby in zone B is promoted by the runbook within 15 minutes, no second writer accepts a commit, and a known order remains readable." Steady state: the synthetic purchase succeeds, and the capture SLI does not accumulate UNKNOWN beyond its threshold. Blast radius: staging, or a production test tenant if you have earned that level. Abort: any write success against the old endpoint after fence, or error budget on the real capture path if you were reckless enough to run this against production money. You have not earned production data-destroying experiments. Stopping one zone of stateless app servers is a different, smaller experiment and still needs consent.

The Principles of Chaos document is the ethic: hypothesize, vary real-world events, automate what you have already done manually, minimize blast radius. A tool that randomly kills production nodes on the first afternoon violates the last principle. Sequence is the design.

Practical Implementation or Design Walkthrough

Problem statement

Write the DR and continuity plan for TechLearningPro orders and payment ledger. Catalog and analytics are in the table only to show they do not inherit the ledger's RTO. You have one region today, multi-AZ, nightly snapshots in-region, and an untested replica.

Requirements, assumptions, constraints

Business limits, agreed with product and finance for this exercise:

• Acknowledged captures must not be lost to a single zone failure. RPO 0 inside that fault.
• Regional loss: RPO 5 minutes, RTO 60 minutes, because the measured full restore is too slow and a warm replica is the chosen spend.
• Bad migration: able to return to a point in the last 14 days. RTO 4 hours, because PITR of this volume was measured at 3 hours in the assumption set.
• Manual sales ceiling: 50 orders, then the storefront shows "purchases paused."

Assumptions you would replace with measurement: ledger 500 GB and growing 5 GB a week; WAL about 20 GB a day; cross-region lag under 5 seconds when healthy; DNS TTL 60 seconds; on-call can start the runbook within 10 minutes of a page.

Constraints: PAN is not in the ledger, only acquirer references; backups must not be deletable for 14 days by the application account; tenant predicates remain after restore; two writers are a failed recovery even if both are "up."

The plan

Ordinary zone loss. Synchronous standby in a second zone. Runbook Z: fence, promote, repoint the application, confirm with the synthetic purchase. Target 15 minutes. This path does not use the backup.

Region loss. Asynchronous replica in region B, warm, with apps deployed and workers off. Runbook R: confirm region A is the problem from an external synthetic, fence any reachable A writer, promote B, enable workers in B only, shift DNS, reconcile captures with the acquirer, post the status sentence. Target 60 minutes. If replication lag is above 5 minutes at decision time, the decider announces the larger RPO before shifting traffic.

Logical corruption. Runbook P: PITR to a new instance in a clean network, validate, fence the bad primary, cut traffic, replay deletions and acquirer reconciliation. Target 4 hours. Continuity script runs from minute 15 of this clock, not from minute 240.

Account compromise. Backups live in a separate account with a 14-day lock. Application credentials cannot delete them. Restoration uses a break-glass role two people must approve. This path is slow on purpose.

Continuity script

At T+15 minutes of a declared disaster, support switches to the macro: "If you saw a confirmation, your access will return when we restore. Do not pay again. If you did not see a confirmation, you have not been charged, or the charge is being matched — we will email you." That macro is only true if the idempotency design is true. It is part of the plan because the payment case study made it true.

Finance opens the manual sheet, capped at 50, with columns: email, course id, amount, external reference, idempotency key. Engineering loads that sheet after restore through the same capture API, not through a direct insert that skips the ledger rules.

The status page is updated by the decider, not by the health check.

Validation queries after promote

A known order id from before the incident is present. A cross-tenant read with tenant A's role returns no tenant B row. Exactly one node reports as primary. The outbox worker count in the fenced region is zero. The synthetic purchase in the test tenant completes and reconciles. These are pass/fail. A "looks fine" message in chat is not the gate.

Game day calendar

• Monthly: timed restore of the base backup to an isolated instance. Record the duration. If it exceeds 70 percent of the PITR RTO, the RTO is already false.
• Quarterly: staging zone failover, including fence proof and a deliberate attempt to write to the old primary, which must fail.
• Twice a year: staging region cutover and failback by rebuilding A from B, not by resurrecting A's old disk.
• After every destructive migration rehearsal: one PITR practice.

Production chaos is limited to stopping a single stateless order-api pod while the SLO is watched. Database promotion in production is a real failover, done when needed or in a planned window with product consent, not a surprise injection.

Expected result

An ADR records warm regional standby, immutable 14-day WAL, and the RTO table. The in-region-only snapshot is kept as a convenience and removed from the DR slide so nobody calls it the plan. The first game day fails, because the DNS TTL was 3600 seconds in the real zone despite the runbook saying 60. That failure is the deliverable. The TTL is fixed, the runbook gains a pre-check, and the RTO clock is believed only after the next game day measures it.

Enterprise Real-World Scenarios

The replica that replicated the delete

The Thursday migration ran ALTER TABLE in a way that dropped a column the new code no longer understood, and a backfill deleted rows it should have archived. The streaming replica in the other zone was current, so promotion would have promoted the bug. The on-call's first instinct was to fail over because the runbook's only branch was failover. A staff engineer who had written the RPO table stopped them: this fault is logical, the control is PITR, and failing over spends the disruption without moving the recovery point.

The snapshot was six hours old and in the same region. WAL archiving had been "planned." There was no archive. The six-hour snapshot became the recovery point by force, and the RPO that day was six hours, not zero. Reconciliation with the acquirer restored the captures that had a settlement record. Orders that were acknowledged and not yet settled, inside a buggy window, needed the manual sheet. Forty-two learners were on it. The cap of fifty would have held. Seventy would have been a pile of unkeyed spreadsheets.

Two changes followed. WAL archiving to a locked bucket in another account was turned on that week, and the next game day proved a restore to a chosen minute. The runbook gained a branch at the top: "Is the data wrong, or is the node dead?" Dead nodes promote. Wrong data restores to a time. The health check was changed so a synthetic order's failure pages even when both Postgres nodes answer SELECT 1. The incident review refused a new active-active project. The hole was the missing archive, and a second writer would have doubled it.

Region evacuation and the failback that almost resurrected the corpse

A provider region went read-only for three hours. The warm replica in region B was 8 seconds behind. The decider announced an 8-second RPO, fenced region A by stopping the instances through the provider API, promoted B, and shifted DNS. Checkout returned in 40 minutes, inside the 60-minute RTO. Receipts duplicated for nine minutes because the outbox worker in A had not been scaled to zero before DNS moved, and a few messages had already been claimed. The template's idempotent wording kept that from becoming a support crisis. The runbook's worker step had been after the DNS step. The order was wrong. It was corrected to: fence, stop workers everywhere, promote, start workers only in the survivor, then DNS.

Failback the next day was the near miss. A's volumes recovered with old data and a primary role in their local configuration. A script labeled "restore traffic to home region" would have pointed the application at a stale primary and split history. The engineer running the script asked for the LSN check because the quarterly game day had included that question. A was rebuilt as a replica of B and caught up before anyone considered a new promotion. The home region did not become primary again until the following window. Product wanted the latency back immediately. The runbook's failback rule held because it was written as a prohibition — do not promote a disk that diverged — and not as a suggestion.

The privacy residual showed up in the same week. A learner deletion from the morning of the outage was missing after the timeline wobble, and the backup would have kept the row anyway for the 14-day lock. The deletion log was replayed. Counsel's notice already said backups linger. The incident made that sentence operational. DR and the threat model finally shared a number.

Best Practices

• Set RPO and RTO per service with the person who accepts the harm. Do not inherit the strictest number for every store, and do not invent one number for "the platform."
• Compare the RTO to the error budget. A human regional failover does not support a 99.99 percent monthly SLO. Say which fault model the SLO covers.
• Use replication for node and zone loss. Use immutable PITR backups for bad writes and deleted history. Use a separate account for credential loss.
• Put restore duration, detection, and traffic shift inside the RTO. Time them.
• Fence before promote. Prove a single writer. Plan failback as "rebuild the replica," not "turn the old primary on."
• Reconcile payments with the acquirer after any timeline change. Replay through idempotency keys.
• Write the continuity script, the manual ceiling, and the status-page owner. Test the script.
• Match backup retention to detection time and to the privacy deletion residual.
• Run game days on a calendar, with a steady state, a hypothesis, and an abort. Start with restore tests.
• Keep workers single-homed across a failover. Duplicate outbox consumers are a second incident.
• Record the choice in an ADR with a measured number that reopens it.
• After every real event, fix the runbook's order before you add a new region.

Common Mistakes and Anti-Patterns

Replication called a backup. The replica is current and the only copy. A bad migration or a dropped table is instantly in both places. It happens because lag dashboards look like safety. The repair is WAL archiving with a retention longer than detection, and a branch in the runbook for logical faults.

Backups in the same failure domain. The snapshot account is the application account, in the same region. A region loss or a stolen key removes the primary and the history. The repair is another region, a separate identity, and a lock window.

Untimed RTO. The document says 30 minutes and the first restore takes 11 hours. It happens because size was used as a proxy for duration. The repair is a measured restore on a calendar, and an RTO that matches the measurement until a warm standby changes it.

Failover without fencing. The old primary still accepts writes from a connection pool. It happens because DNS was updated and felt sufficient. The repair is a fence with proof, short-lived database sessions, and a validation that the old endpoint rejects writes.

Active-active on a ledger by default. Two regions take captures because a slide showed two clouds. Conflicts become double seats. The repair is a warm replica or a tenant-partitioned single writer. Last-write-wins is not a payment strategy.

Health check as the decider. SELECT 1 stays green while the schema is wrong. The repair is a synthetic journey and the capture SLI as inputs to the declaration.

Failback to a stale disk. The home region returns and a script points traffic at it. The repair is a written prohibition and an LSN comparison. Rebuild from the survivor.

Chaos as surprise. Random production kills with no hypothesis and no abort. It happens because a tool was installed to look mature. The repair is the principles: steady state, hypothesis, blast radius, and no data-destroying production injections until the restore test is boring.

Troubleshooting and Design Review

Symptom: Failover completed and learners see old data, or two different totals.

Likely cause: split brain, a stale failback, or a replica promoted while lag was large and the RPO announcement never happened.

Investigation: compare LSNs and the promotion timestamps. Look for writes on both endpoints in the database logs.

Corrective action: fence one side, choose the timeline that matches acquirer settlements, rebuild the other as a replica. Do not merge money rows by timestamp.

Symptom: The RTO was met for the database and missed for the product.

Likely cause: DNS TTL, cold application caches, workers still in the dead region, or secrets for region B missing.

Investigation: walk the runbook's clock against the incident timeline. The gap after "primary promoted" is the clue.

Corrective action: add those steps to the timed game day. Pre-stage secrets and images. Fix TTL in the real zone file, then verify with a query, not with the runbook text.

Symptom: PITR restored the data and resurrected deleted learners.

Likely cause: the recovery target predates legitimate deletions, and the deletion log was not replayed.

Investigation: compare the deletion log to the target time.

Corrective action: replay deletions before opening traffic. If you cannot tell legitimate deletions from the attack, keep the storefront in read-only mode until a human reconciles.

Symptom: Backup storage cost is high and someone wants to drop retention to two days.

Likely cause: a real bill, and no detection-time argument on the page.

Investigation: ask when the last quiet corruption would have been noticed. Ask what the privacy residual requires. Those are different numbers; the retention must satisfy the stricter one that is still an obligation, and you should show both.

Corrective action: keep the detection-based retention or fund faster detection. Do not shorten below the privacy lock without counsel.

Symptom: Every game day is cancelled because enrollment week is close.

Likely cause: there is no calendar owner, so DR tests lose to features.

Investigation: look at the last successful timed restore.

Corrective action: monthly isolated restores are small enough to keep. Treat a skipped quarter as a failed control in the same way a missed patch is a failed control.

Design-review questions: What is the RPO and RTO of checkout, and who accepted them? Which fault does replication cover, and which fault requires PITR? Where is the backup, in which account, locked for how long? What was the last measured restore duration? How do you fence, and how do you prove one writer? What is the failback prohibition? How are in-flight captures reconciled? What does support say at minute fifteen? What is the game day's steady state and abort?

Interview Questions and Answers

Q1. What is the difference between RPO and RTO?

RPO is how much data you can afford to lose, measured as time: an RPO of five minutes means the recovered world may miss the last five minutes of commits, and you need a way to find them, often by reconciling with a system of record such as the acquirer. RTO is how long you can afford to be unrestored, from detection and decision through promote, traffic shift, and a correctness check. A fast promote of an empty database meets neither. A perfect backup that takes eleven hours to restore misses a one-hour RTO even though the RPO is fine. I state both per service.

Q2. Why is a replica not a backup?

A replica's job is to converge to the primary, including the primary's mistakes: a dropped column, a bad update, a malicious delete performed with valid credentials. That is what you want for a dead node and what you do not want for logical corruption. A backup retained past the detection delay, ideally with point-in-time WAL replay and an immutability lock, is the control for those faults. I want both, in the runbook as separate branches: dead node promotes, wrong data restores to a time.

Q3. How do you design regional recovery for an order ledger without active-active writes?

I keep a single writer in region A, a warm replica in region B applying WAL asynchronously, and applications deployed in B with workers off. RPO is the lag, published and alerted, with a number product accepted — in the walkthrough, five minutes. RTO is a measured sum of detection, fence, promote, worker move, and DNS. I do not let both regions take captures. If a tenant needs to live in a nearer region, I partition writers by tenant home region so each row still has one writer. Last-write-wins on a money table is not an alternative I put on the slide.

Q4. What has to be true before you promote a standby?

The old primary must be fenced so it cannot accept writes, and I need proof: the instance is stopped, or a write attempt fails, not merely that DNS changed. Then I promote one node and confirm it is the only primary. Then I start background workers only on the survivor. Then I shift traffic. Then I run a known-order read and a cross-tenant deny. In-flight payments are reconciled with the acquirer by idempotency key before anyone replays them. Skipping the fence is how a partition becomes two ledgers.

Q5. A 99.99 percent SLO and a 30-minute regional RTO are both in the packet. What do you say?

They describe different promises and currently contradict each other. Four nines in a 30-day time-based budget is about 4.3 minutes, and a 30-minute regional event consumes many budgets. I split the fault model. The SLO can cover instance and zone loss with automatic multi-AZ failover inside a few minutes. Regional loss is a disaster objective with its own RTO, reported separately, and it will blow the monthly error budget when it happens. If product wants the SLO to include region loss, the architecture has to get regional recovery inside 4 minutes, which is a different and expensive design. I will not leave both sentences unqualified.

Q6. How do you test a DR plan?

I time a restore from backup on a calendar, in an isolated network, and I record the duration against the RTO. I run staging failovers that include a fence and a failed write to the old primary. I practice PITR to a minute before a bad migration. I practice failback by rebuilding the old region as a replica. Production experiments stay inside a blast radius that cannot destroy the ledger; stopping one stateless pod is enough until the restore test is dull. Each exercise has a steady state taken from the SLO, a hypothesis, and an abort. An untested document is not a plan.

Q7. What belongs in business continuity that is not in the failover steps?

Who declares the disaster, what support says, when the status page changes and who may change it, the manual order ceiling and the idempotency keys on that sheet, how instructors and enterprise customers are told, and when you stop selling. Those procedures run during the RTO, not after it. I test them. A perfect promote with a lying status page and an unbounded spreadsheet is a business incident wrapped around a successful failover.

Q8. How do backups interact with a privacy deletion promise?

The threat model should already have accepted a residual: deletions do not propagate into immutable backups until the lock expires. The DR retention and the privacy notice need the same number. Operationally, a point-in-time restore can resurrect a deleted person, so I replay a deletion log before traffic returns. I also keep the backup account away from application credentials and away from casual support copies. A restore is a new trust boundary full of personal data.

Q9. Walk through an in-flight capture during failover.

If the ledger has a terminal state, I believe it and I do not charge again. If the state is UNKNOWN or the client never received a response, I query the acquirer with the original idempotency key and write the terminal state that comes back. I never "reprocess the last hour" as new payments. Entitlements are derived from the reconciled ledger, so a database that lost thirty seconds of WAL can be repaired from settlements. That is how a nonzero technical RPO can still meet a user-visible promise of no lost payment, and it only works if the key was stored before the call.

Q10. What is wrong with using last-write-wins when both regions accepted orders during a partition?

The later timestamp can be the wrong capture: a smaller amount, a duplicate seat, or a refund clobbering a payment. Clocks are not an order of business events. I would rather have stopped writes on one side than merge them. If a partition already created two histories, the repair is a human reconciliation against the acquirer, not an automatic winner. The design that avoids the question is single-writer, with fencing, so the partition cannot create the second history.

Q11. What would you measure in the first game day of a plan that has never been executed?

The restore of last night's base backup, end to end, into a network that cannot route to production, and the wall clock beside the documented RTO. I would also read the real DNS TTL and the real replication lag, because documents drift. I would not start by killing a production primary. The first game day's job is to replace fantasy durations with measurements and to discover missing WAL, missing secrets in region B, and a runbook step in the wrong order.

Hands-On Exercises

Exercise: DR plan for an order platform

Problem statement. TechLearningPro's order and payment ledger runs in one region, multi-AZ, with a nightly snapshot stored next to the database and a replica you have never promoted. Product wants purchases to survive a zone loss without lost acknowledgements, and to return within an hour of a region loss. Write the plan. You may tell product the hour is impossible until a specific control exists.

Tasks.

1. Build an RPO and RTO table for capture, entitlement, catalog, and analytics. Name who accepts each row.
2. Map fault to control: zone loss, region loss, bad migration, stolen admin key.
3. Draw the topology and the failover sequence, including the fence and the worker step.
4. Specify backup location, retention, immutability, and how restore time enters the RTO. Label any number you have not measured.
5. Write the continuity paragraph: declaration, support macro, manual ceiling, status page.
6. Explain in-flight capture reconciliation.
7. Schedule game days and write one experiment with steady state, hypothesis, blast radius, and abort.
8. Note one privacy constraint the restore must honor.

Expected deliverables. The table, the diagram, the runbook outline, the experiment, and an ADR-sized decision if you reject active-active or the current snapshot design.

Evaluation rubric.

A one-page statement that "we fail over to the replica" with no RPO, no fence, and no timed restore is not a DR plan.

Lesson Summary

Disaster recovery returns a named system inside an RPO and an RTO that a business owner accepted. Business continuity is what the company says and does before that return finishes. Replication, including the multi-AZ synchronous standby from lesson 29, survives a dead node and a dead zone and faithfully copies a bad migration. Immutable point-in-time backups in another account survive the bad migration and a stolen application credential. A warm replica in a second region is the order platform's regional design; a second writer is not, unless each row still has a single home.

The RTO includes detection, fencing, promotion, workers, DNS, and a correctness check. Numbers you have not timed are assumptions. Game days exist to delete those assumptions, starting with a restore into an isolated network. In-flight money is reconciled with the acquirer through idempotency keys. Failback rebuilds a replica. It does not resurrect a stale primary.

Architecture Performance and Capacity Review closes this module's practice set. A recovery design that cannot carry the enrollment-week load after failover has only recovered the diagram. The capacity review asks whether the survivor was ever sized to be alone.

Further Reading

• PostgreSQL documentation on continuous archiving and point-in-time recovery — base backups, WAL archives, and recovery_target_time, the portable mechanism this lesson uses.
• PostgreSQL documentation on high availability and synchronous_commit — the zone-level RPO mechanism, taught operationally in lesson 29.
• Beyer, Jones, Petoff, and Murphy, Site Reliability Engineering — testing disaster tolerance and the habit of treating an untested failover as unproven.
• Principles of Chaos Engineering — steady state, hypothesis, real-world events, automation, and minimized blast radius.
• NIST SP 800-34 — contingency planning vocabulary for continuity versus recovery. Use it for the plan's table of contents, not as a template to fill with adjectives.
• AWS Well-Architected reliability pillar, and the equivalent Azure and Google reliability guides — questions about restoration and regional isolation. Answer them with the measured restore, not with a service list.
• This course: Database Replication and Consistency, CAP Theorem and Consistency Models, Non-Functional Requirements and SLO Design, Idempotency and Exactly-Once Processing, Architecture Security and Threat Modeling, Design a Payment Processing System, Load Balancing and Traffic Management.