Incident Management and SRE Fundamentals

AWS Cloud & DevOps Engineering · Lesson 40 · Observability, Security and Reliability

Lesson Overview

Telemetry without agreements is noise. Site Reliability Engineering (SRE) fundamentals give TechLearningPro a language for reliability: SLIs, SLOs, and error budgets, plus a human operating system for incidents—roles, severity, runbooks, communications, and blameless postmortems. The Monday board is not “up” because an EC2 status check is green; it is up when learners can complete the journeys you promised at the reliability you documented.

This lesson is intentionally tool-light on proprietary ticketing brands and heavy on principles you can implement with CloudWatch alarms (lesson 36), traces (38–39), and your organization’s chat/incident process. You will write an SLO, an alert policy aligned to burn, an incident runbook, and a postmortem template for a progress API failure.

Prior observability lessons matter; Deployment Strategies: Blue-Green, Canary and Rolling supplies release context; Well-Architected operational excellence supplies culture. Tag documentation Project=tlp-lesson40. No need to leave expensive resources running—most deliverables are documents plus alarm references.

Learning Objectives

1. Define SLI, SLO, and SLA and keep them distinct in conversation.
2. Choose user-centric SLIs for TechLearningPro journeys (availability, latency, correctness).
3. Set SLOs that are ambitious but achievable; derive error budgets.
4. Explain error budget policy: when to slow feature work versus stabilize.
5. Design multi-window burn-rate style alerting concepts without requiring a specific vendor.
6. Define incident severity levels with examples for education SaaS.
7. Assign incident roles: incident commander, tech lead, communications, scribe.
8. Write a runbook that on-call can execute under stress.
9. Practice incident communications to faculty/learners without oversharing internals.
10. Run a blameless postmortem with concrete action items and owners.
11. Avoid vanity SLOs and pager storms that destroy response quality.
12. Connect SLOs to CloudWatch metrics you can actually measure.
13. Relate error budgets to canary speed and freeze windows.
14. Distinguish urgent incident work from problem management follow-up.
15. Deliver SLO + alert policy + runbook + postmortem artifacts.

Prerequisites

• Lessons 36–39 for signals you will bind SLIs to.
• Lesson 34 deployment strategies for release risk context.
• AWS Well-Architected Framework and Cost Basics.
• Willingness to measure before setting ambitious “four nines” theater.

Detailed Core Concepts

SLI — what you measure

A Service Level Indicator is a quantitative measure of some aspect of service. Good TechLearningPro SLIs: proportion of progress-save requests that succeed within 300ms at the API edge; proportion of Monday board loads under a latency threshold; correctness checks for grade finalization jobs. Bad SLIs: CPU average, disk is fine, “we feel good.”

SLO — what you promise internally

A Service Level Objective targets an SLI over a window (for example 99.9% successful progress saves over 28 days). SLOs are engineering contracts. They should be visible on a Monday reliability board.

SLA — what you promise contractually

A Service Level Agreement may include credits and legal terms. Not every SLO is an SLA. Do not casually rename them in executive meetings.

Error budgets

If SLO is 99.9% availability over a month, the error budget is the allowed 0.1% failure. When budget is healthy, ship features and accept change risk. When budget is burned, prioritize reliability work, slow releases, or freeze risky changes per policy. Error budgets turn reliability from slogan into math.

Alerting on symptoms and burn

Page humans when user journeys burn budget quickly (fast burn) or steadily (slow burn)—conceptually multi-window burn-rate alerting popularized by Google SRE. Implement with CloudWatch metric math and alarms as your tooling allows; do not invent a non-existent CloudWatch “SLO object” unless AWS documents one you choose to use. Many teams compute SLI metrics and alarm on rates/thresholds that approximate burn policy.

Incident lifecycle

Detect → triage/severity → page → mitigate (restore service) → communicate → resolve → postmortem → track actions. Mitigation beats root-cause purity during the incident. Root cause belongs in the postmortem.

Severity

Example education SaaS severities (adapt to your org):

• SEV1: widespread learners cannot learn/pay/submit; data loss risk.
• SEV2: major feature impaired (progress sync failing for many).
• SEV3: limited impact / workaround exists.
• SEV4: minor / cosmetic.

Severity drives response time and communications—not egos.

Roles

Incident Commander (IC) coordinates; Tech Lead drives technical mitigation; Communications updates stakeholders; Scribe timelines facts. On small teams one person wears multiple hats—still name the hats so nothing is forgotten.

Blameless postmortems

Focus on systemic causes: missing alarms, unclear ownership, risky deploy process, inadequate testing—not “engineer X is careless.” Blameless is not consequence-free for willful policy violations; it is a method to get truth so the system improves. Every postmortem needs owners and due dates—not “we should be more careful.”

Architecture and Design Diagrams

mermaidCopy
flowchart TB
  Users[Learners journeys]
  SLI[SLI metrics]
  SLO[SLO + error budget]
  Alert[Burn / symptom alerts]
  Inc[Incident process]
  PM[Blameless postmortem]
  Improve[Reliability work / slower change]

  Users --> SLI --> SLO --> Alert --> Inc --> PM --> Improve --> SLI

Incident swimlane:

mermaidCopy
sequenceDiagram
  participant A as Alarm
  participant IC as Incident Commander
  participant TL as Tech Lead
  participant C as Comms
  A->>IC: page
  IC->>TL: mitigate
  IC->>C: status updates
  TL->>IC: mitigated
  IC->>IC: resolve + schedule PM

Technical Explanation

Choosing SLIs for progress API

Availability SLI: non-5xx (and non-timeout) ratio for POST /progress. Latency SLI: fraction under threshold. Quality SLI: canaries that save and read back progress. Measure at a consistent point (API Gateway/ALB metrics + synthetic canary). Document exclusions (load test accounts) carefully so budgets are not fiction.

Setting thresholds with data

Use historical CloudWatch metrics before picking 99.99%. If you only achieve 99.5% today, an honest 99.5% SLO with a reliability roadmap beats a fake four nines.

Alert policy sketch

• Page on fast burn of availability SLO or hard symptom (elevated 5xx for N minutes during traffic).
• Ticket on slow burn without waking people at 3 a.m. if policy allows.
• Do not page on CPU alone.

Runbook contents

Symptoms, severity guide, dashboards/traces to open, mitigation steps (rollback canary, scale, disable feature flag), escalation graph, comms templates, what not to do (random restart everything).

Postmortem template

Summary, impact (users, duration, SLO burn), timeline, root causes (systemic), what went well, what went poorly, action items with owners/dates, detection gaps.

Linking to AWS tooling

CloudWatch alarms and dashboards (36), CloudTrail for “who changed what” (37), X-Ray/OTel for hop latency (38–39). Incident tools can be chat + tickets; AWS Systems Manager Incident Manager exists as a capability—evaluate against current docs if your org wants AWS-native incident tooling, without requiring it for this lesson’s deliverables.

Practical Implementation or Design Walkthrough

Problem

Write an SLO, alert policy, incident runbook, and postmortem for TechLearningPro progress API / Monday board reliability.

Steps

1. Pick one journey; draft SLIs with metric sources from lesson 36.
2. Set a 28-day SLO; compute error budget mathematically.
3. Write alert policy mapping to alarms you can implement.
4. Author runbook with rollback pointing to lesson 34 mechanisms.
5. Simulate a tabletop incident; fill a postmortem with fake-but-realistic timeline.
6. Present Monday board view: SLO target, current attainment, budget remaining.
7. ADR on error budget policy (feature freeze rules).

Acceptance evidence

Four artifacts + tabletop notes + links to metrics/alarms.

Enterprise Real-World Scenarios

SLO on CPU

Nobody trusts reliability. Fix: user journey SLIs.

Page on everything

On-call resigns. Fix: severity + burn-based paging.

Blameful postmortem

Silence next time. Fix: blameless facilitation; systemic actions.

No IC

Everyone debugs; no comms. Fix: role cards.

Error budget ignored

SLO is wallpaper. Fix: release policy tied to budget.

Best Practices

1. User-centric SLIs.
2. Honest SLO targets.
3. Visible error budgets.
4. Symptom and burn alerting.
5. Clear severities.
6. Named incident roles.
7. Runnable runbooks.
8. Blameless postmortems with owners.
9. Track actions to completion.
10. Practice tabletops before exam week.
11. Tie deploys to budget.
12. Separate mitigate vs deepen.
13. Customer-ready status language.
14. Review SLOs quarterly.
15. Keep artifacts next to the service repo.

Common Mistakes and Anti-Patterns

1. SLA/SLO/SLI confusion.
2. Vanity four nines.
3. Alerting on causes only.
4. No error budget policy.
5. Blame culture.
6. Runbooks outdated.
7. Comms radio silence.
8. Action items without owners.
9. Measuring only in staging.
10. Treating SRE as “the ops team alone.”

Troubleshooting and Design Review

Interview Questions and Answers

Q1: SLI vs SLO vs SLA? A: Measure vs internal target vs contractual promise.
Q2: Error budget? A: Allowed unreliability derived from SLO.
Q3: Why user-centric SLIs? A: Infra green ≠ users successful.
Q4: Burn-rate alerting idea? A: Page when budget consumes too fast.
Q5: Incident Commander? A: Coordinates response and decisions.
Q6: Mitigate vs root cause? A: Restore first; analyze after.
Q7: Blameless? A: Seek systemic truth without fear-driven silence.
Q8: Severity purpose? A: Match response/comms to impact.
Q9: Runbook quality bar? A: Executable by tired humans.
Q10: Monday board? A: Show SLO attainment and budget.
Q11: Canary link? A: Spend budget knowingly; halt when burning.
Q12: Action items? A: Owned, dated, reviewed.
Q13: Tooling? A: CloudWatch etc. measure; process makes SRE.
Q14: Too many SLOs? A: Start with few critical journeys.
Q15: Fake SLO risk? A: Erodes trust—measure honestly.

Hands-On Exercises

Exercise: SLO, alert policy, incident runbook, and postmortem

Deliver the four artifacts for progress API / Monday board.

Challenge ADR

Product wants to ship daily during a burned error budget. Write the policy decision and communication plan.

Lab evidence checklist

SLI definitions with metric names, SLO math, alert mapping, runbook, postmortem from tabletop, tags in doc headers.

Narrative: exam week SEV1

Progress saves fail for 40 minutes. IC rolls back canary; comms posts status; postmortem finds missing dependency throttle alarm and a runbook gap; actions close in a week; budget policy pauses non-critical launches two days.

Continuity

Next: AWS KMS and Encryption Architecture—reliability includes protecting data with sound key design.

Error budget policy one-pager

Include: calculation window, freeze thresholds, who can grant exceptions, how budget interacts with security patches (usually still ship), and reporting cadence to the Monday board.

Comms templates

Draft SEV1 external status (“Progress saving degraded—retry shortly”) and internal bridge notes. Avoid speculative root cause in public.

Tabletop facilitation guide

IC opens; scribe starts timeline; inject “DynamoDB throttles” after five minutes; require a rollback decision; end with PM assignment. Forty minutes calendar invite beats unread wiki.

Extended practice scenario 1 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: write availability and latency SLIs for progress save journey

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 2 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: compute 28-day error budget from a chosen SLO target

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 3 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: map burn-style alert intent onto CloudWatch alarms you can build

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 4 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: draft SEV definitions with education SaaS examples

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 5 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: runbook rollback steps referencing blue/green or canary controls

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 6 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: blameless postmortem with three owned action items

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 7 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: Monday board mock showing SLO attainment and budget remaining

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 8 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: tabletop incident with IC and comms roles exercised

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 9 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: write availability and latency SLIs for progress save journey

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 10 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: compute 28-day error budget from a chosen SLO target

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 11 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: map burn-style alert intent onto CloudWatch alarms you can build

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 12 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: draft SEV definitions with education SaaS examples

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 13 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: runbook rollback steps referencing blue/green or canary controls

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 14 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: blameless postmortem with three owned action items

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 15 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: Monday board mock showing SLO attainment and budget remaining

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 16 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: tabletop incident with IC and comms roles exercised

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 17 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: write availability and latency SLIs for progress save journey

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 18 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: compute 28-day error budget from a chosen SLO target

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 19 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: map burn-style alert intent onto CloudWatch alarms you can build

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 20 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: draft SEV definitions with education SaaS examples

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 21 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: runbook rollback steps referencing blue/green or canary controls

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 22 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: blameless postmortem with three owned action items

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

SLO math worked example

Suppose progress-save availability SLO is 99.9% over 30 days. Allowed downtime-equivalent error budget is 0.1% of requests (not always wall-clock—use request-based budgets when traffic varies). If you serve 10,000,000 progress saves in the window, the budget is about 10,000 failed saves. A bad canary that fails 50,000 requests has already overdrawn—feature work pauses per policy until reliability actions restore budget or leadership explicitly accepts the risk in writing.

Show this math on the Monday board beside current failed-count and remaining budget. Engineers understand numbers better than vibes.

Alerting anti-pattern clinic

• Alerting on every dependency’s CPU: pages without user impact.
• Single threshold with no duration: flaps.
• Same page for staging and prod: alert fatigue.
• No link to dashboard: wasted minutes.
• No owner: tickets rot.

Rewrite one noisy alert set as a classroom exercise using only symptom SLIs.

Postmortem action taxonomy

Classify actions as: detect (new alarm), mitigate faster (runbook/rollback automation), prevent (test, guardrail, quota), and educate (training). Aim for at least one detect and one prevent item each SEV1/SEV2. “Be more careful” is not an action.

Faculty and learner communications

Education platforms owe clear status language: which feature, workaround (retry, use mobile later), next update time. Never blame a named engineer publicly. Coordinate with support so ticket macros match the status page. Lesson 37’s CloudTrail may inform internal root-cause discussions; it does not belong in the public banner.

Extended practice scenario 23 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: request-based error budget spreadsheet for progress saves

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 24 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: rewrite a noisy alert into a symptom SLI alarm

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 25 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: postmortem action taxonomy with detect and prevent items

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 26 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: public status template reviewed by a non-engineer classmate

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 27 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: error budget freeze exception memo exercise

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 28 — lesson 40 Incident management SRE

In a timed drill, implement or design the core lab for lesson 40 (Incident management SRE) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: request-based error budget spreadsheet for progress saves

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Lesson Summary

SRE fundamentals make TechLearningPro reliability explicit: SLIs that mirror learner journeys, SLOs and error budgets that drive release decisions, alerts that respect human attention, incident roles that restore service quickly, and blameless postmortems that change the system. Wire them to CloudWatch and traces you already built.

Next: AWS KMS and Encryption Architecture.

Further Reading

• Google SRE Book — Service Level Objectives
• Google SRE Workbook — Alerting on SLOs
• AWS Well-Architected Operational Excellence
• Amazon CloudWatch Alarms
• AWS Systems Manager Incident Manager