AWS Cloud & DevOps Tutorial 0/70 lessons ~6 min read Lesson 68“IAM, VPC, EC2, Lambda, ECS, EKS, Terraform, CI/CD, and AWS operations”

Project 5: Observability, SRE and Disaster Recovery

Projects 1–4 gave TechLearningPro platforms without proving you can **see, page, recover, and learn**. This capstone operations project overlays **CloudWatch metrics/logs/alarms**,

Course progress0%

Focus

15 guided sections

Practice signal

Examples included

Career prep

Interview Q&A included

Project 5: Observability, SRE and Disaster Recovery

AWS Cloud & DevOps Engineering · Lesson 68 · Production Projects and Interview

Lesson Overview

Projects 1–4 gave TechLearningPro platforms without proving you can see, page, recover, and learn. This capstone operations project overlays CloudWatch metrics/logs/alarms, tracing concepts (X-Ray/OpenTelemetry), SLIs/SLOs/error budgets, incident runbooks, backup and disaster recovery drills, and a blameless postmortem onto the three-tier app, container CI/CD path, serverless orders, and landing-zone baselines. The portfolio proof is a failure simulation with telemetry screenshots, recovery timeline, and postmortem—not a wallpaper of green dashboards.

Continuity: CloudWatch (36), tracing (38–39), incidents/SRE (40), backup/DR (45), HA (46), FinOps (47), PRRs (63), Projects 64–67. Tag Project=tlp-lesson68.

Learning Objectives

Define SLIs/SLOs/error budgets for at least one prior project workload.
Build a dashboard covering golden signals for that workload.
Wire actionable alarms (symptom-based) to SNS/ChatOps.
Add tracing or correlation IDs across at least one request path.
Write an incident runbook with severity, roles, and first actions.
Choose backup/PITR/DR strategy aligned to RPO/RTO targets.
Execute a failure injection (process kill, bad deploy, region dependency deny, or queue poison).
Recover within a measured time; capture timeline.
Produce a blameless postmortem with action items.
Estimate observability and DR cost dimensions illustratively.
Document multi-account telemetry aggregation ideas from lesson 67.
Avoid pager fatigue via SLO-aligned alerts.
Link PRR checklist items to live evidence.
Package portfolio: dashboards, alarms, drill evidence, postmortem, ADRs.
Explain warm standby vs backup/restore trade-offs without fake RTO numbers.
Test restore, not only backup existence.
Separate security detections (GuardDuty) from availability paging where appropriate.
Include log retention and PII redaction considerations.
Connect FinOps: idle dashboards/log ingest costs.
Prepare a five-minute SRE interview walkthrough.

Prerequisites

At least one prior project stack still deployable or rebuildable.
Lessons 36, 40, 45 strongly recommended.
Budget for log ingest, alarms, and any secondary-region DR experiment.

Detailed Core Concepts

Business problem

Exam week: enrollment API latency spikes and order DLQ fills. Without SLOs, on-call pages on CPU cosmetics; without DR drills, backups are fiction. Leadership asks “are we up?” and the answer is Slack guesswork.

Functional requirements

Documented SLIs (e.g., availability of /health and successful order accept rate).
SLO with error budget policy (e.g., illustrative 99.9% monthly—pick a number and justify, do not claim AWS SLA as your SLO).
Dashboard + alarms.
Runbook + escalation.
Backup/restore or multi-AZ evidence.
One game-day failure with postmortem.

Non-functional requirements

Alert actionable within five minutes of symptom; recovery steps executable by a peer; cost of telemetry bounded; no secrets in logs; destroyable lab alarms/subscriptions.

Service selection rationale

Need	AWS / practice	Why
Metrics/logs/alarms	CloudWatch	Native, integrates with AWS resources
Tracing	X-Ray or OTel→backend	Latency decomposition
Notifications	SNS → email/Slack webhook patterns	Page humans
Backups	AWS Backup, RDS snapshots, DynamoDB PITR	Managed restore paths
DR patterns	Backup/restore, pilot light, warm standby	Match RTO/RPO
Process	Incident commander + postmortem	Organizational reliability

Repo structure

text
tlp-sre-dr/
  observability/terraform/   # dashboards as code if supported; alarms; SNS
  runbooks/
  slos/
  gamedays/
  postmortems/
  docs/adr/
  evidence/
  README.md

IaC approach

Alarms and SNS in Terraform/CDK alongside app stacks; dashboards as JSON/source where maintainable. Avoid clickops-only monitoring for portfolio credibility.

Security

Least-privilege for incident roles; encrypt SNS where required; redact tokens from logs; CloudTrail for who changed alarms.

CI/CD

Synthetic canary (CloudWatch Synthetics—verify current APIs) optional in pipeline smoke; do not page on deploy flakes without classification.

Monitoring design

Golden signals: latency, traffic, errors, saturation. For serverless: DLQ depth, throttles. For ECS: unhealthy hosts, 5xx. For landing zone: CloudTrail delivery failures.

DR design

Define RPO/RTO as targets, measure restore. DynamoDB PITR for orders; RDS snapshots for three-tier; ECR images immutable for compute recovery. Cross-Region only if NFR demands—and budget allows.

Cost

Log ingest and retention dominate; high-cardinality metrics hurt; secondary region warm standby costs real money. Prefer sampling and retention tiers.

Architecture and Design Diagrams

mermaid
flowchart TB
  Workloads[Projects 64-66 workloads]
  CW[CloudWatch metrics/logs]
  Trace[X-Ray / OTel]
  Dash[Dashboard]
  Alarm[Alarms]
  SNS[SNS]
  Human[On-call]
  Backup[AWS Backup / PITR]
  DR[Restore drill]

  Workloads --> CW --> Dash
  Workloads --> Trace
  CW --> Alarm --> SNS --> Human
  Workloads --> Backup --> DR
  Human --> Runbook[Runbooks]
  Human --> PM[Postmortem]

mermaid
flowchart LR
  SLI[SLI measurement] --> SLO[SLO]
  SLO --> Budget[Error budget]
  Budget -->|Burn slow| Feature[Ship features]
  Budget -->|Burn fast| Freeze[Change freeze / reliability work]

mermaid
sequenceDiagram
  participant F as Failure inject
  participant S as System
  participant A as Alarm
  participant IC as Incident commander
  participant R as Restore/rollback
  F->>S: Break dependency
  S->>A: Symptom metric
  A->>IC: Page
  IC->>R: Execute runbook
  R->>S: Recover
  IC->>IC: Postmortem timeline

Technical Explanation

Writing an SLO

Pick user-visible success: “proportion of authenticated enrollment API requests completing < 500 ms with non-5xx.” Measure with ALB metrics or custom metrics. Error budget = 100% − SLO. When budget is burned, prioritize reliability.

Alarm hygiene

Alert on symptoms (success rate, latency, DLQ) not only causes (CPU). Require runbook URL in alarm description. Use composite alarms thoughtfully (confirm current CloudWatch features in docs).

Game day ideas (pick one primary)

ECS bad task definition → unhealthy hosts → rollback.
Lambda notify poison → DLQ → redrive.
Disable subnet route / SG rule in lab → detect → fix.
Delete object in non-prod and restore from backup.
Never game-day production without authorization.

Postmortem fields

Summary, impact, timeline, root contributing factors, what went well, what went poorly, action items with owners/dates. Blameless language.

Multi-account

Security account may aggregate findings; workload accounts own SLOs. Cross-account dashboard strategies vary—document approach; verify Console/Observability features currently available rather than inventing federation APIs.

Linking PRR

Lesson 63 PRR checklist items “alarms exist,” “restore tested,” “runbook reviewed” become green only with evidence from this project.

Practical Implementation or Design Walkthrough

Problem

Run a failure simulation and present telemetry, recovery, and postmortem for a prior TechLearningPro platform.

Steps

Budget; tag Project=tlp-lesson68.
Choose workload (recommend Project 2 or 3).
Write SLO doc; implement dashboard+alarms via IaC.
Attach SNS email for lab; verify test notification.
Ensure backups/PITR enabled as applicable; perform restore in lab.
Execute game day; record timestamps.
Recover; verify SLO burn chart narrative.
Write postmortem; file action items (e.g., missing alarm on DLQ).
ADR for DR pattern; cleanup extra regions/alarms if needed.

Enterprise Real-World Scenarios

Pager fatigue

CPU alarms every deploy. Fix: symptom SLOs; silence windows with care.

Backup never restored

Ransomware tabletop fails. Fix: quarterly restore drills with evidence.

Trace without sampling policy

Bill explodes. Fix: sampling rules; retention.

Multi-region fantasy

Claiming active-active without data plane design. Fix: honest pilot light ADR.

Postmortem blame

Engineer named; learning dies. Fix: blameless charter from lesson 40/58 culture.

Best Practices

SLOs before fancy dashboards.
Symptom-based paging.
Runbooks linked from alarms.
Test restores.
Game days on a calendar.
Blameless postmortems with owners.
Bound log volume.
Tag telemetry resources.
Align DR pattern to RTO/RPO.
Multi-AZ first; multi-region when justified.
Error budget policy published.
Synthetic checks for critical journeys.
Separate security ops from availability where noisy.
IaC for alarms.
Cleanup lab paging endpoints.

Common Mistakes and Anti-Patterns

Dashboard theater without alarms.
Alerting on causes only.
Untested backups.
Inventing RTO numbers as guarantees.
Secrets in logs.
No incident commander.
Infinite retention.
Skipping postmortem actions.
Paging every 4xx.
Claiming multi-region without failover test.

Troubleshooting and Design Review

Symptom	Cause	Action
No pages during outage	Wrong metric/threshold	Align to SLI
Constant pages	Threshold too tight	Use burn rates / loosen
Restore slow	Cold backup pattern	Revisit DR tier
Missing traces	Not instrumented	Add middleware/SDK
High CloudWatch bill	Verbose logs	Filter; retention
Confusion across accounts	No ownership map	Define telemetry owners

Interview Questions and Answers

Q1: SLI vs SLO vs SLA? A: Indicator measured; objective target; agreement with users/business—don't confuse with AWS SLAs.
Q2: Error budget? A: Allowed unreliability before reliability work takes priority.
Q3: Golden signals? A: Latency, traffic, errors, saturation.
Q4: Why symptom alerts? A: Users care about success, not CPU alone.
Q5: DR patterns? A: Backup/restore, pilot light, warm standby, multi-site—cost vs RTO.
Q6: Prove backups? A: Restore drill evidence.
Q7: Postmortem? A: Blameless learning with actions.
Q8: Game day? A: Controlled failure to test detection/recovery.
Q9: Tracing value? A: Find latency bottlenecks across services.
Q10: Cost control? A: Retention, sampling, avoid high cardinality.
Q11: Link to projects? A: Overlay on 64–66; accounts from 67.
Q12: PRR? A: Evidence feeds readiness.
Q13: Multi-AZ vs region? A: AZ for common failures; region for regional events—justify.
Q14: On-call? A: Runbooks, escalation, IC role.
Q15: Portfolio proof? A: SLO, dashboard, alarm page, restore, postmortem.

Deep dive: Mapping SLIs to prior projects

Project 1 (three-tier): SLI = successful HTML/API responses from ALB with latency under a chosen threshold; saturation via RDS connections / CPU.
Project 2 (ECS): SLI = healthy host count ratio and 5xx rate; deploy failure rate as a supporting indicator.
Project 3 (serverless orders): SLI = accept success rate; DLQ depth as a leading reliability indicator for notify path.
Project 4 (landing zone): SLI-like operational checks = CloudTrail delivery success, Config recorder status, budget alert pipeline health—platform reliability enables application reliability.

Write one SLO document per critical user journey, not one vague “99.99% everything” claim. Interviewers punish unmeasured nines.

Deep dive: Sample postmortem outline (fill with your game day)

text
Title: Enrollment API elevated 5xx during game day
Date / authors:
Impact: users, duration, error budget burned
Timeline (UTC):
Detect → Diagnose → Mitigate → Resolve
Contributing factors (systems, not people):
What went well:
What went poorly:
Action items: owner, due date, tracking link
Appendix: dashboard screenshots, alarm ARNs, restore IDs

Deep dive: Cost of not practicing DR

Untested snapshots create false confidence. A two-hour restore discovery during a real incident costs more than a scheduled lab restore. Record wall-clock restore time as the only RTO evidence that matters for your portfolio.

Deep dive: On-call handoff checklist

Before a shift: verify paging path with a test alert in lab or agreed channel; skim open error-budget burn; confirm runbook links resolve; know break-glass path for the landing zone; know who is secondary. After a shift: note lingering risks, unfinished postmortem actions, and any alarm that fired without a runbook. TechLearningPro treats handoff quality as a reliability control equal to autoscaling.

Hands-On Exercises

Exercise: Failure simulation with telemetry, recovery, and postmortem

Deliverable: Complete SRE/DR pack overlaid on a prior project, with cost+cleanup notes.

Challenge ADR

Accept pilot-light DR for orders for 90 days due to budget; schedule warm-standby revisit with measured RTO evidence.

Lab evidence checklist

SLO doc, dashboard, alarm notification proof, restore evidence, game-day timeline, postmortem, ADR; budget/cleanup confirmation.

Narrative: A silent DLQ during exam week taught TechLearningPro that CPU green boards lie; SLO on accept success and DLQ depth fixed paging

A silent DLQ during exam week taught TechLearningPro that CPU green boards lie; SLO on accept success and DLQ depth fixed paging.

Continuity

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Deep dive: ADR template (TechLearningPro)

text
# ADR-NNNN: Title
Date:
Status: Proposed | Accepted | Deprecated | Superseded by ADR-XXXX
Context:
Decision:
Consequences:
Evidence / links:

Deep dive: Portfolio README skeleton

text
TechLearningPro Project — Lesson 68
- Problem / NFRs
- Architecture (diagram + service rationale)
- Repo layout
- Deploy / destroy
- Security checklist
- Observability
- Failure tests
- Cost notes (illustrative; verify pricing)
- ADRs

Deep dive: Budget and tagging

Create or verify a budget with email/SNS subscriber before any apply. Tag Project=tlp-lesson68, Owner=<you>, Environment=lab. Screenshot Cost Explorer filtered by tag after destroy to prove idle cost returned near zero (allow for delayed billing).

Extended enterprise scenario: acquisition pressure

Leadership wants the lab promoted as “production-ready marketing copy.” Refuse without PRR evidence from lesson 63, WAR risks from lesson 58, and failure-test artifacts from this lesson.

Extended enterprise scenario: disputed decision

Two senior engineers disagree on the primary architecture choice. Run a timeboxed RFC, record ADR with revisit date, and keep the rejected alternative documented so interviews and audits see the trade-off—not silence.

Extended practice scenario 1 — 68 observability sre and disaster recovery

Confirm AWS Budgets alerts exist for the sandbox; tag every resource Project=tlp-lesson68. Do not proceed without a budget subscriber.

In a timed drill for Observability SRE and disaster recovery at TechLearningPro, implement or design the core lab with a partner observing. The observer's job is to stop any step that invents API flags, hardcodes secrets, invents pricing numbers presented as current quotes, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: SLO doc, dashboard, alarm notification proof, restore evidence, game-day timeline, postmortem, ADR, and destroy confirmation for any lab resources.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps become flashcards. Premium courses finish when you can operate under time pressure without treating Observability SRE and disaster recovery as a mystery.

Extended practice scenario 2 — 68 observability sre and disaster recovery

Document the lesson's primary architecture decision in a ten-line ADR: context, decision, status, consequences.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 3 — 68 observability sre and disaster recovery

Map identity paths: who can create/modify the resources in this lesson; which IAM roles are least privilege.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 4 — 68 observability sre and disaster recovery

Design observability: CloudWatch metrics/alarms or equivalent signals that prove the control works.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 5 — 68 observability sre and disaster recovery

Run or design a negative test: denied access, misconfiguration, or failed dependency—capture evidence.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 6 — 68 observability sre and disaster recovery

Estimate cost dimensions for this lesson's activities. Label figures illustrative; verify pricing pages with dates.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 7 — 68 observability sre and disaster recovery

Produce a cleanup checklist and execute destroy for any billable lab resources created for lesson 68.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 8 — 68 observability sre and disaster recovery

Cross-link prior course lessons that this design depends on; list two failure modes inherited from those layers.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 9 — 68 observability sre and disaster recovery

Security review: encryption at rest/in transit, public exposure checks, CloudTrail visibility for management events.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 10 — 68 observability sre and disaster recovery

Reliability review: failure injection, rollback path, how detection and recovery work.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 11 — 68 observability sre and disaster recovery

Performance review: throughput, latency, or scaling limits relevant to Observability SRE and disaster recovery at TechLearningPro.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 12 — 68 observability sre and disaster recovery

Write a runbook section for on-call: symptoms, first three diagnostic steps, escalation, rollback.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 13 — 68 observability sre and disaster recovery

Config/compliance angle: which controls must be true before production promotion.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 14 — 68 observability sre and disaster recovery

Multi-account consideration: how this workload sits in landing-zone OU boundaries from lessons 48–49 and 67.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 15 — 68 observability sre and disaster recovery

Interview drill: explain Observability SRE and disaster recovery at TechLearningPro to a hiring manager in three minutes without inventing APIs.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 16 — 68 observability sre and disaster recovery

Pair review: partner stops any invented CLI flags, hardcoded secrets, or fake current prices.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 17 — 68 observability sre and disaster recovery

FinOps continuity from lesson 47: idle resources, data transfer, and when to decommission lab stacks.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 18 — 68 observability sre and disaster recovery

DR continuity from lesson 45 and 68: recoverability must survive the change you are shipping.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 19 — 68 observability sre and disaster recovery

Produce evidence pack: diagrams, configs, test results, cleanup proof.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 20 — 68 observability sre and disaster recovery

Challenge constraint: tighter monthly budget—what do you sequence differently?

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 21 — 68 observability sre and disaster recovery

Challenge constraint: stricter compliance—what evidence gates promotion?

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 22 — 68 observability sre and disaster recovery

Challenge constraint: shorter maintenance window—what pre-staging changes?

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 23 — 68 observability sre and disaster recovery

Teach-back: recite the Lesson Overview paragraph, then check against the text; flashcard gaps.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 24 — 68 observability sre and disaster recovery

Dress rehearsal: tabletop under a timer with observer notes.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Extended practice scenario 25 — 68 observability sre and disaster recovery

Final synthesis note: one paragraph connecting Observability SRE and disaster recovery at TechLearningPro to TechLearningPro production ownership.

Caps Projects 64–67 and feeds interview lessons 69–70 with war stories and diagrams you can whiteboard.

Compliance evidence pack for auditors / platform leads

Budget configs, tagging proof (Project=tlp-lesson68), architecture diagram, ADR, negative-test evidence, encryption settings notes, CloudTrail/Config references where applicable, pricing-page retrieval dates for any dollar claims, cleanup confirmation.

Glossary drill

Review every bold term introduced in Detailed Core Concepts and Interview Q&A. Write one sentence definitions from memory; correct against the lesson. Unfamiliar AWS console navigation becomes a lab note, not a guess. Focus terms: SLI, SLO, error budget, RTO, RPO, game day, blameless postmortem.

Budget and cleanup sworn statement

I verified budgets remain enabled with alert subscribers. I deleted or scheduled deletion for orphan lab resources from lesson 68 named ____. I labeled every savings or cost figure with pricing-page retrieval dates or marked it illustrative. Signed ________ date ________.

Interview story to memorize

Craft a 60-second story: TechLearningPro needed Observability SRE and disaster recovery; you chose a design with explicit trade-offs; you proved it with tests; you governed cost with budgets and tags; you left an ADR so the next engineer does not reverse the decision blindly. Tell the game-day story with timestamps.

Self-assessment rubric

Area	Weight
Conceptual accuracy and service selection	20%
Working design/lab and reproducibility	25%
Security and least-privilege	20%
Reliability, observability, troubleshooting	20%
Documentation, ADR, cleanup	15%

Final synthesis before summary

Mastery of Observability SRE and disaster recovery at TechLearningPro is not logo recognition—it is owning failure modes, cost meters, identity boundaries, and the written decision that keeps TechLearningPro coherent under change.

Extended facilitation script (print for guild leads)

Open with exam-week outage. Ask what SLI failed, who was IC, and whether restore was ever practiced. Require postmortem actions in git. Close by assigning three interview prompts for lesson 69 drawn from this drill.

Lesson Summary

Observability, SRE, and DR turn TechLearningPro’s prior projects into operable systems: SLOs drive alerts, game days prove recovery, and postmortems create change—not blame. Tag Project=tlp-lesson68.

Next: AWS Cloud and DevOps System Design Interviews.