AWS Cloud & DevOps Tutorial 0/70 lessons ~6 min read Lesson 45“IAM, VPC, EC2, Lambda, ECS, EKS, Terraform, CI/CD, and AWS operations”

Backup, Disaster Recovery and Resilience

High availability inside one Region (next lesson) does not survive Region-wide impairment, account-level sabotage, or ransomware that encrypts production data. **Disaster recovery

Course progress0%

Focus

15 guided sections

Practice signal

Examples included

Career prep

Interview Q&A included

Backup, Disaster Recovery and Resilience

AWS Cloud & DevOps Engineering · Lesson 45 · Observability, Security and Reliability

Lesson Overview

High availability inside one Region (next lesson) does not survive Region-wide impairment, account-level sabotage, or ransomware that encrypts production data. Disaster recovery (DR) is the disciplined ability to restore TechLearningPro’s teaching capability within agreed RTO (recovery time objective) and RPO (recovery point objective). AWS Backup centralizes backup policies across supported services (EBS, RDS, DynamoDB, EFS, and others—verify current support matrix), vaults, lifecycle, and cross-account/Region copy patterns. Architecture strategies—backup and restore, pilot light, warm standby, and multi-site active/active—trade cost for recovery speed. This lesson forces honest objectives for learner progress data versus cacheable media, then maps them to AWS mechanisms without inventing replication APIs.

Cross-links: AWS KMS and Encryption Architecture, Amazon CloudWatch Metrics, Logs and Alarms, Incident Management and SRE Fundamentals. Tag Project=tlp-lesson45. Budget for backup storage, cross-Region copy, and restore-test compute; delete restored labs promptly.

Learning Objectives

Define RPO and RTO with business-tier examples for TechLearningPro.
Compare backup/restore, pilot light, warm standby, and multi-site strategies.
Configure AWS Backup plans, vaults, and lifecycle policies conceptually and in a lab.
Encrypt backups with KMS and plan cross-account decrypt roles.
Design cross-Region copy for critical recovery points.
Run a tabletop DR exercise with DNS, secrets, and identity dependencies.
Measure restore time empirically toward RTO validation.
Distinguish multi-AZ HA from multi-Region DR.
Apply immutability concepts (vault lock) for ransomware resilience per current docs.
Position S3 versioning/replication versus AWS Backup for object data.
Estimate DR cost drivers; verify pricing; avoid always-on warm standby vanity.
Segregate backup vs restore IAM duties.
Document failback as carefully as failover.
Troubleshoot failed jobs and missing recovery points.
Clean up restore debris and expired lab backups.
Prepare interview answers on strategy selection.
Map DR to Reliability pillar Well-Architected questions.
Preview FinOps tension: resilience spend vs waste (lesson 47).

Prerequisites

Multi-AZ mental model (deepened in lesson 46); KMS; Route 53 alias/failover awareness from lesson 12.
Budget alert; sandbox; no production destructive tests without approval.

Detailed Core Concepts

RPO and RTO

RPO is how much data loss (time) you can tolerate. RTO is how long services may be down/degraded. Progress database might target RPO minutes and RTO under an hour for paid cohorts; marketing pages might accept hours. Write targets before buying architecture.

Strategy spectrum

Backup and restore: cheapest steady state; slowest RTO—restore from vaults/snapshots, rebuild compute with IaC. Pilot light: core data replicated; minimal compute “pilot” in DR Region; scale out on disaster. Warm standby: scaled-down but running environment; faster RTO; higher cost. Multi-site active/active: serve traffic from multiple Regions continuously; hardest data consistency; highest cost/complexity—only with explicit conflict design.

AWS Backup

Backup plans assign schedules and lifecycle (transition to cold storage where supported). Vaults store recovery points. Resource assignment via tags or explicit selections—TechLearningPro should tag Backup=true / BackupTier=gold. Monitor Backup job events with EventBridge/CloudWatch.

Consistency and application awareness

Crash-consistent vs application-consistent backups matter for databases. Prefer service-native snapshot integrations Backup orchestrates; quiesce apps when docs require. Test restores—untouched backups are Schrödinger’s recovery.

People and process

DR fails when only one engineer knows the runbook. Tabletop exercises, game days, and on-call rotations make DR real. Status communications belong in the plan.

Architecture and Design Diagrams

mermaid
flowchart LR
  Prod[Primary Region workloads]
  Plan[AWS Backup plan]
  Vault[Backup vault KMS]
  Copy[Cross-Region copy]
  DRVault[DR Region vault]
  Restore[Restore and IaC scale]
  DNS[Route 53 failover]

  Prod --> Plan --> Vault --> Copy --> DRVault --> Restore --> DNS

mermaid
flowchart TB
  subgraph strategies [Cost vs RTO]
    BR[Backup restore]
    PL[Pilot light]
    WS[Warm standby]
    AA[Active active]
  end
  BR -->|faster RTO / higher cost| PL --> WS --> AA

Technical Explanation

Selecting strategy for TechLearningPro tiers

Tier	Example	Typical pattern (illustrative)
Gold	Progress DB, grades	Pilot light or warm standby + tight RPO
Silver	Lesson metadata	Backup/restore + frequent backups
Bronze	Marketing static	S3 versioning + CloudFront; rebuild

Exact choice is an ADR with Finance.

Cross-Region and cross-account

Copy recovery points to a DR Region/account. KMS keys must allow decrypt in the restore role’s account. Test permissions before disaster day.

Vault Lock / immutability

Where AWS Backup Vault Lock (or equivalent immutability controls currently offered) is enabled, recovery points resist deletion—valuable against ransomware and rogue admins. Understand governance modes and compliance modes from current docs before enabling on labs (can be hard to undo).

DNS cutover

Route 53 health checks and failover policies shift clients. Lower TTLs before drills. Document who can change DNS and how to fail back without split-brain writes.

Performance of restores

RTO must include: detect outage, declare DR, restore data, apply IaC, warm caches, validate, switch DNS, communicate. Measure each phase in game days.

Security

Encrypt backups; restrict vault IAM; monitor DeleteRecoveryPoint APIs via CloudTrail; separate duties.

Cost

Backup storage, selective copy, warm standby compute 24/7, data transfer—verify pricing. Warm standby for everything bankrupts education startups; tier ruthlessly.

Practical Implementation or Design Walkthrough

Problem

Write a DR plan and conduct a tabletop recovery exercise for TechLearningPro progress data.

Steps

Budget; tags Project=tlp-lesson45.
Draft RPO/RTO per tier.
Implement or design Backup plan + vault for a lab database/volume.
Perform a restore test; time it.
Tabletop pilot light vs warm standby decision with cost sketch (non-binding figures).
ADR including failback and ransomware notes.
Cleanup restored resources and unneeded recovery points.

Enterprise Real-World Scenarios

Untested backups

Restore fails due to KMS policy. Game day finds it; fix roles; retest.

Region event

Pilot light scales; DNS failover; progress API RTO met; video CDN already global via CloudFront—media RTO near zero if origins intact.

Ransomware

Immutable vaults retain clean points; restore to isolated account; forensic scrub.

Overbought active/active

Team claims multi-Region writes without conflict resolution; data diverges. Roll back to warm standby until CQRS/conflict design exists.

Finance cuts Backup

After quiet year, backups disabled; next human error loses grades. Tie Backup to compliance evidence (lesson 44).

Best Practices

Write RPO/RTO before architecture.
Tier workloads gold/silver/bronze.
Use AWS Backup with tag-based assignment.
Encrypt with KMS; plan DR decrypt.
Cross-Region copy for gold tier.
Quarterly restore tests with clocks.
Tabletop DNS, secrets, IdP, artifacts.
Immutable vaults for critical data where appropriate.
Separate backup/restore roles.
Document failback.
Align HA (multi-AZ) and DR (multi-Region) clearly.
Budget warm standby consciously.
Automate Backup monitoring alarms.
Include container images and IaC state in dependency lists.
Communicate learner impact honestly.

Common Mistakes and Anti-Patterns

Snapshots never restored.
Same Region only for “DR.”
RTO fantasies without IaC timing.
Single human hero runbooks.
Unencrypted backups.
Warm standby for bronze tiers.
Ignoring secrets/DNS in DR.
Deleting recovery points freely without vault controls.
Skipping failback planning.
Inventing cross-Region features for services that lack them.

Troubleshooting and Design Review

Symptom	Cause	Action
Backup job failed	IAM/KMS/service limits	Fix role; read job error
Restore too slow	Cold data / size	Revise RTO or strategy
Cannot decrypt	KMS policy	Grant DR role
DNS sticky	High TTL	Lower TTL pre-event
Split-brain	Dual writers	Fence primary

Interview Questions and Answers

Q1: RPO vs RTO? A: Data loss tolerance vs downtime tolerance.
Q2: Pilot light? A: Minimal DR footprint scaled out on disaster.
Q3: Warm standby? A: Continuously running reduced-capacity DR environment.
Q4: AWS Backup? A: Central policy-based backup across supported AWS resources.
Q5: Multi-AZ vs DR? A: AZ resilience vs Region/account disaster recovery.
Q6: Why test restores? A: Untested backups are assumptions.
Q7: Vault encryption? A: KMS-protected recovery points; key policy for DR.
Q8: Active/active risk? A: Data conflicts without design.
Q9: Ransomware angle? A: Immutable/offline copies; isolated restore.
Q10: Cost control? A: Tier strategies; verify pricing; don’t warm-standby everything.
Q11: Tag-based backup? A: Assign resources via Backup selection tags.
Q12: Failback? A: Planned return to primary without corruption.
Q13: S3 DR? A: Versioning, replication, or Backup where supported—match needs.
Q14: Game day? A: Scheduled exercise validating people+tech.
Q15: First DR declaration steps? A: Confirm blast radius, choose runbook, communicate, restore/scale, cut over, validate.

Hands-On Exercises

Exercise: DR plan + tabletop

Deliverable: RPO/RTO table, strategy ADR, Backup evidence or detailed design, timed restore or simulated timeline, tabletop notes, cleanup.

Challenge ADR

Leadership demands active/active everywhere. Counter with tiered pilot light/warm standby and data consistency prerequisites.

Lab evidence checklist

Budget; tags; vault/plan; restore timing; tabletop attendees; cleanup.

Narrative: lost grades

Human drops table; Backup restore within RTO; postmortem adds IAM protections and change windows.

Continuity

Next: High Availability and Fault-Tolerant Architecture for in-Region failure domains and graceful degradation—the everyday resilience DR complements.

Deep dive: dependency mapping workshop

List every dependency to serve one quiz submit: Route 53, CloudFront, WAF, ALB, ECS service, RDS, Redis, Secrets Manager, KMS, IAM Identity Center token validation, third-party email. Highlight which are Region-scoped. DR plans that restore only RDS fail the quiz submit path.

Deep dive: RPO math

If backups run every 6 hours without continuous replication, worst-case RPO approaches 6 hours plus failure timing. Continuous change-data replication can tighten RPO—confirm the specific AWS database feature you rely on. Never claim “zero RPO” without architecture that streams commits offsite.

Deep dive: restore testing automation

Schedule a monthly restore into an isolated account using Automation documents; destroy after validation. Alarms if job fails. This is Reliability pillar practice, not optional homework.

Extended practice scenario 1 — 45 backup disaster recovery and resilience

Write RPO/RTO targets for three TechLearningPro tiers: marketing site, progress API+DB, video assets in S3. Justify numbers with business impact, not vanity.

In a timed drill for RPO RTO and DR strategies, implement or design the core lab with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, console or CLI output, and destroy/disable confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, KMS encryption (lesson 41), multi-AZ design (lesson 46), CloudWatch alarms, and CI promotion rules. List two failure modes you simulated, including detection and fix. Estimate cost drivers and teardown timing.

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps become flashcards. Premium courses finish when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 2 — 45 backup disaster recovery and resilience

Create an AWS Backup plan (or detailed design) for lab EBS/RDS/DynamoDB as available—verify resource support in current Backup docs. Tag Project=tlp-lesson45.

Extended practice scenario 3 — 45 backup disaster recovery and resilience

Map backup vault encryption to a KMS key; document who can decrypt in DR account.

Extended practice scenario 4 — 45 backup disaster recovery and resilience

Tabletop: primary Region unavailable. Walk pilot light bring-up steps with roles and DNS cutover criteria.

Extended practice scenario 5 — 45 backup disaster recovery and resilience

Tabletop: warm standby. Compare monthly cost sketch vs pilot light using illustrative ratios only; verify real prices separately.

Extended practice scenario 6 — 45 backup disaster recovery and resilience

Test restore of a sample volume or database snapshot to a disposable instance; measure elapsed time toward RTO.

Extended practice scenario 7 — 45 backup disaster recovery and resilience

Cross-Region copy design for Backup vaults; discuss compliance data residency constraints.

Extended practice scenario 8 — 45 backup disaster recovery and resilience

Define backup frequency vs RPO; show why daily backups cannot meet 5-minute RPO.

Extended practice scenario 9 — 45 backup disaster recovery and resilience

Continuous replication options (e.g., RDS read replica / Aurora global themes at conceptual level)—verify services you claim.

Extended practice scenario 10 — 45 backup disaster recovery and resilience

Chaos: delete a lab table after backup; restore; prove data freshness vs RPO.

Extended practice scenario 11 — 45 backup disaster recovery and resilience

Runbook: ransomware suspicion—immutable backup vaults / vault lock concepts per current AWS Backup Vault Lock docs.

Extended practice scenario 12 — 45 backup disaster recovery and resilience

S3 versioning + replication vs Backup for objects—when each applies for lesson media.

Extended practice scenario 13 — 45 backup disaster recovery and resilience

Game day calendar: quarterly restore test ownership between platform and data teams.

Extended practice scenario 14 — 45 backup disaster recovery and resilience

Cost worksheet: Backup storage, restore testing compute, cross-Region transfer—verify pricing pages.

Extended practice scenario 15 — 45 backup disaster recovery and resilience

Multi-account: backups in workload account vs centralized backup account patterns.

Extended practice scenario 16 — 45 backup disaster recovery and resilience

IAM least privilege for backup operators vs restore operators (segregation of duties).

Extended practice scenario 17 — 45 backup disaster recovery and resilience

Document dependencies: DNS, secrets, container images, IAM Identity Center—DR is more than the database.

Extended practice scenario 18 — 45 backup disaster recovery and resilience

Pilot light checklist: minimal ASG capacity, AMI freshness, IaC apply time budgeted into RTO.

Extended practice scenario 19 — 45 backup disaster recovery and resilience

Warm standby checklist: traffic shift via Route 53 health checks; failback plan.

Extended practice scenario 20 — 45 backup disaster recovery and resilience

Multi-site active/active tease: only where data conflict design exists; otherwise forbid fantasy.

Extended practice scenario 21 — 45 backup disaster recovery and resilience

Align DR tier with lesson 46 HA: multi-AZ is not multi-Region DR.

Extended practice scenario 22 — 45 backup disaster recovery and resilience

Communications template for DR declaration and learner-facing status.

Extended practice scenario 23 — 45 backup disaster recovery and resilience

Evidence: restore screenshot, timing log, vault settings, cleanup of restored resources.

Extended practice scenario 24 — 45 backup disaster recovery and resilience

Cleanup: delete restored resources, expire old lab recovery points, budget remains.

Extended practice scenario 25 — 45 backup disaster recovery and resilience

Full dress rehearsal tabletop with stopwatch and scribe.

Compliance evidence pack for auditors

Backup plan IDs, vault KMS, restore test logs with timestamps, tabletop minutes, exception list for systems without Backup coverage.

Glossary drill

RPO; RTO; recovery point; backup vault; pilot light; warm standby; active/active; failback; vault lock; cross-Region copy; crash-consistent; game day.

Budget and cleanup sworn statement

Restored resources destroyed; lab recovery points lifecycle applied; budget on; Cost Explorer check scheduled for Backup line items.

Interview story to memorize

“We set gold-tier RPO/RTO for TechLearningPro progress data, used AWS Backup with cross-Region copy, proved restore timing in a game day, chose pilot light over blanket warm standby for cost, and documented DNS and KMS failback.”

Extended technical explanation: application-consistent database backups

For relational progress stores, prefer backup mechanisms integrated with the engine’s snapshot facilities as orchestrated by AWS Backup or native snapshot APIs documented for RDS/Aurora. Application-consistent backups may require brief fencing or transaction flush procedures owned by the DBA runbook—coordinate with feature flags that pause grade writes during scheduled freeze windows when the business allows. Document whether your chosen approach is crash-consistent only; if so, practice recovery with DBCC/engine check tools appropriate to your engine after restore. Never assume a snapshot equals a tested, queryable gradebook without running the validation queries in the game day script.

Extended technical explanation: cold storage lifecycle trade-offs

Backup plans can transition recovery points to colder storage classes where AWS Backup supports them for the resource type. Cold storage lowers monthly cost and can raise restore latency—directly threatening RTO if gold-tier restores unexpectedly pull from cold. Policy pattern: keep recent daily points warm for gold tiers; transition older points cold for compliance retention. Verify transition rules and restore times in current AWS Backup documentation before promising executives “seven-year retention with one-hour RTO.”

Module continuity paragraph

Lesson 44 proved configurations stay compliant; lesson 45 proves data can return after disaster; lesson 46 proves everyday AZ failures do not become disasters. FinOps lesson 47 asks whether your warm standby is earning its keep. Keep these narratives connected in architecture reviews so resilience does not sprawl into unexamined spend.

Hands-on narrative: stopwatch restore script

Create a markdown stopwatch log with columns: declare_DR, vault_restore_started, vault_restore_finished, iac_apply_finished, smoke_test_passed, dns_changed, learners_notified. Fill it during the tabletop even if some rows are simulated. The habit of measuring phases exposes that DNS and communication often dominate wall-clock RTO more than raw snapshot restore. Bring the filled log to architecture review; it ends arguments based on feelings.

Segregation of duties drill

Write two IAM policy sketches (high-level actions only, verify exact actions in docs before apply): BackupOperator can create plans and copy recovery points but cannot restore to production subnets; RestoreOperator can restore into DR account only. Explain why combining them risks ransomware helpers who delete live data and backups together.

Budget and cleanup sworn statement

I verified the budget alert remains enabled. I deleted restored lab databases, volumes, and temporary compute used for restore tests named ____. I applied lifecycle or deleted unneeded lab recovery points in vault ____. I will re-check Cost Explorer within three days for AWS Backup and related transfer charges. Signed ________ date ________.

Final synthesis before summary

Resilience is objectives, copies, tested restores, practiced people, and honest spend. Untested DR is fiction.

Lesson Summary

Backup and DR give TechLearningPro recoverable futures: define RPO/RTO, pick backup/restore vs pilot light vs warm standby vs active/active deliberately, orchestrate with AWS Backup and KMS, test restores, and tabletop the human path. Tag Project=tlp-lesson45; budget storage and DR compute; clean up restore labs.

Next: High Availability and Fault-Tolerant Architecture.

Backup, Disaster Recovery and Resilience

Backup, Disaster Recovery and Resilience

Lesson Overview

Learning Objectives

Prerequisites

Detailed Core Concepts

RPO and RTO

Strategy spectrum

AWS Backup

Consistency and application awareness

People and process

Architecture and Design Diagrams

Technical Explanation

Selecting strategy for TechLearningPro tiers

Cross-Region and cross-account

Vault Lock / immutability

DNS cutover

Performance of restores

Security

Cost

Practical Implementation or Design Walkthrough

Problem

Steps

Enterprise Real-World Scenarios

Untested backups

Region event

Ransomware

Overbought active/active

Finance cuts Backup

Best Practices

Common Mistakes and Anti-Patterns

Troubleshooting and Design Review

Interview Questions and Answers

Hands-On Exercises

Exercise: DR plan + tabletop

Challenge ADR

Lab evidence checklist

Narrative: lost grades

Continuity

Deep dive: dependency mapping workshop

Deep dive: RPO math

Deep dive: restore testing automation

Extended practice scenario 1 — 45 backup disaster recovery and resilience

Extended practice scenario 2 — 45 backup disaster recovery and resilience

Extended practice scenario 3 — 45 backup disaster recovery and resilience

Extended practice scenario 4 — 45 backup disaster recovery and resilience

Extended practice scenario 5 — 45 backup disaster recovery and resilience

Extended practice scenario 6 — 45 backup disaster recovery and resilience

Extended practice scenario 7 — 45 backup disaster recovery and resilience

Extended practice scenario 8 — 45 backup disaster recovery and resilience

Extended practice scenario 9 — 45 backup disaster recovery and resilience

Extended practice scenario 10 — 45 backup disaster recovery and resilience

Extended practice scenario 11 — 45 backup disaster recovery and resilience

Extended practice scenario 12 — 45 backup disaster recovery and resilience

Extended practice scenario 13 — 45 backup disaster recovery and resilience

Extended practice scenario 14 — 45 backup disaster recovery and resilience

Extended practice scenario 15 — 45 backup disaster recovery and resilience

Extended practice scenario 16 — 45 backup disaster recovery and resilience

Extended practice scenario 17 — 45 backup disaster recovery and resilience

Extended practice scenario 18 — 45 backup disaster recovery and resilience

Extended practice scenario 19 — 45 backup disaster recovery and resilience

Extended practice scenario 20 — 45 backup disaster recovery and resilience

Extended practice scenario 21 — 45 backup disaster recovery and resilience

Extended practice scenario 22 — 45 backup disaster recovery and resilience

Extended practice scenario 23 — 45 backup disaster recovery and resilience

Extended practice scenario 24 — 45 backup disaster recovery and resilience

Extended practice scenario 25 — 45 backup disaster recovery and resilience

Compliance evidence pack for auditors

Glossary drill

Budget and cleanup sworn statement

Interview story to memorize

Extended technical explanation: application-consistent database backups

Extended technical explanation: cold storage lifecycle trade-offs

Module continuity paragraph

Hands-on narrative: stopwatch restore script

Segregation of duties drill

Budget and cleanup sworn statement

Final synthesis before summary

Lesson Summary

Further Reading