AWS Cloud & DevOps Tutorial 0/70 lessons ~6 min read Lesson 46“IAM, VPC, EC2, Lambda, ECS, EKS, Terraform, CI/CD, and AWS operations”

High Availability and Fault-Tolerant Architecture

An Availability Zone is a failure domain by design. TechLearningPro learners should not lose the progress API because one AZ’s racks lose power. **High availability (HA)** keeps se

Course progress0%

Focus

15 guided sections

Practice signal

Examples included

Career prep

Interview Q&A included

High Availability and Fault-Tolerant Architecture

AWS Cloud & DevOps Engineering · Lesson 46 · Observability, Security and Reliability

Lesson Overview

An Availability Zone is a failure domain by design. TechLearningPro learners should not lose the progress API because one AZ’s racks lose power. High availability (HA) keeps serving through component and AZ failures using redundant capacity, health-aware load balancing, and data stores that failover. Fault tolerance goes further—continuing without disruption when parts fail—approached asymptotically with careful architecture, never by slogans. This lesson centers multi-AZ design, health checks, retries, timeouts, bulkheads, and graceful degradation: the everyday reliability craft that makes Region-level DR (lesson 45) a rare escalation rather than a weekly fire drill.

You already placed ALBs and Route 53 in Elastic Load Balancing and Route 53 and watched signals in Amazon CloudWatch Metrics, Logs and Alarms. Here you review architectures for SPOFs and prioritize resilience improvements with cost honesty. Tag Project=tlp-lesson46. Budget before creating multi-AZ labs (duplicate compute/databases cost more); clean up when done.

Learning Objectives

Explain AZ failure domains versus Region failure domains.
Design multi-AZ networking and compute placement for learner APIs.
Configure health checks that reflect true dependency health.
Apply retry, backoff, jitter, and idempotency safely.
Set timeout budgets across client, edge, app, and data tiers.
Use bulkheads and queue isolation to contain overload.
Define graceful degradation modes with product agreement.
Distinguish HA from DR and from “more replicas in one AZ.”
Evaluate load balancer target deregistration and connection draining.
Reason about database Multi-AZ failover behavior and client reconnects.
Identify hidden SPOFs (NAT, caches, control-plane assumptions).
Prioritize resilience backlog with risk × likelihood × cost.
Instrument AZ-aware alarms and dashboards.
Troubleshoot flap storms from bad health checks.
Clean up HA lab capacity; leave budgets on.
Prepare interview answers with concrete TechLearningPro examples.
Map patterns to Well-Architected Reliability pillar.
Preview FinOps trade-offs of always-on redundancy (lesson 47).

Prerequisites

Lessons 5, 12, 36, 45 foundations; containers/ECS or EC2 ASG familiarity from Module 2.
Budget alert; staging preferred for chaos-style tests.

Detailed Core Concepts

Failure domains

Power, networking, and control-plane blast radii are scoped. AWS AZs are isolated locations within a Region. Designing across at least two AZs for production stateful and stateless tiers is baseline HA. One AZ with five replicas is still one puddle.

Health checks

ELB/ALB health checks remove bad targets. Deep checks verify DB connectivity or critical dependency; shallow checks only verify the process listens. Deep checks that hit shared DBs can amplify outages—use carefully (cached readiness, separate shallow for LB, deep for orchestration).

Retries, timeouts, idempotency

Retries without backoff create retry storms. Use exponential backoff with jitter. Only retry idempotent operations or those protected by idempotency keys (quiz submit once). Timeouts must be shorter upstream than downstream budgets to fail fast.

Bulkheads

Isolate thread pools, connection pools, and queues so failures in grading do not exhaust the lesson-read API. Cell-based architectures extend this idea at larger scale.

Graceful degradation

When non-critical dependencies fail, serve reduced UX: hide recommendations, serve cached catalogs, disable live chat. Product must pre-approve modes; engineers must wire feature flags and dashboards.

Fault tolerance vs HA language

Marketing “fault tolerant five nines” claims need architecture. Prefer precise: “multi-AZ HA with RPO/RTO X for Region DR.” Honesty builds trust in interviews and executive reviews.

Architecture and Design Diagrams

mermaid
flowchart TB
  Users[Learners]
  ALB[ALB multi-AZ]
  AZ1[Tasks AZ-a]
  AZ2[Tasks AZ-b]
  DBmulti[RDS Multi-AZ / Aurora storage]
  Cache[ElastiCache multi-AZ if used]
  Q[SQS buffer]

  Users --> ALB
  ALB --> AZ1
  ALB --> AZ2
  AZ1 --> DBmulti
  AZ2 --> DBmulti
  AZ1 --> Cache
  AZ2 --> Cache
  AZ1 --> Q
  AZ2 --> Q

mermaid
flowchart LR
  Fail[Dependency failure]
  Detect[Health metrics]
  Degrade[Feature flag degrade]
  Recover[Dependency recovery]
  Full[Full UX]

  Fail --> Detect --> Degrade --> Recover --> Full

Technical Explanation

Multi-AZ compute

ASGs and ECS services span subnets in multiple AZs. Capacity must survive losing one AZ (N+1). Topology spread avoids packing all tasks in one AZ by accident.

Multi-AZ data

RDS Multi-AZ synchronous standby (behavior per engine docs) provides AZ failover. Aurora storage is replicated across AZs by design. ElastiCache and other services have their own HA modes—verify before claiming. Single-node Redis is a classic silent SPOF.

Load balancer behaviors

Deregistration delay / connection draining lets in-flight requests finish. Align application graceful shutdown with LB settings. HTTP keepalive and idle timeouts interact with long uploads—set consciously.

Retry storms and thundering herds

When an AZ fails, surviving capacity sees double traffic plus retries. Load shed, queue, and degrade. Auto Scaling needs headroom or pre-warmed capacity for AZ loss, not only CPU targets from happy days.

Performance

HA can add cross-AZ data transfer latency and cost. Keep chatty pairs co-located when possible; still deploy redundant. Measure; don’t guess.

Security

Redundant does not mean exposed. Security groups and IAM stay least privilege across AZs. Multi-AZ is not an excuse for 0.0.0.0/0.

Cost

Duplicate AZ capacity, Multi-AZ DB pricing, cross-AZ transfer—verify current pricing. Finance conversations need downtime-cost framing from product.

Practical Implementation or Design Walkthrough

Problem

Review an architecture (use your Module 2 lab or a provided SPOF-rich diagram) and prioritize resilience improvements.

Steps

Budget; tag Project=tlp-lesson46.
Inventory failure domains and SPOFs.
Propose top five fixes ranked by risk reduction per dollar.
Implement at least one lab improvement (second AZ tasks, deeper health check, or degradation flag)—or design deeply if account limits.
Define alarms proving the improvement.
ADR; cleanup extra capacity.

Enterprise Real-World Scenarios

Single-AZ “temporary” prod

Temporary becomes eighteen months; AZ event takes exams offline. Fix: multi-AZ mandate for gold tier.

Health check flaps

Deep check hits overloaded DB; all targets marked unhealthy; total outage. Fix: shallow LB check + separate watch.

Retry storm

Mobile clients retry aggressively; remaining AZ melts. Fix: jittered backoff + server load shedding.

No degradation mode

Recommendations service latency pages whole site dependency. Fix: fail-open to static list.

Confused DR declaration

AZ impairment handled by ALB; someone invokes Region DR unnecessarily. Fix: runbooks distinguishing AZ vs Region signals (lessons 45–46).

Best Practices

Multi-AZ for production gold/silver tiers.
Capacity for AZ loss (N+1).
Health checks matched to purpose.
Timeout budgets end-to-end.
Jittered retries + idempotency.
Bulkhead critical paths.
Pre-agreed degradation modes.
Connection draining + graceful shutdown.
AZ-aware dashboards.
Hunt sticky session requirements.
Validate DB failover client behavior.
Avoid single-AZ caches for gold paths.
Chaos/game days in staging.
Document SPOF exceptions with expiry.
Tie HA spend to SLOs.

Common Mistakes and Anti-Patterns

Many replicas one AZ.
Shallow health checks hiding dependency death.
Infinite retries.
Missing timeouts.
Shared thread pool for all outbound calls.
Assuming Multi-AZ DB means zero connection errors on failover.
Ignoring cross-AZ costs until bill shock.
No graceful shutdown.
Feature degradation without product sign-off.
Declaring HA without measuring AZ loss drills.

Troubleshooting and Design Review

Symptom	Cause	Action
Intermittent 5xx one AZ	Bad subnet/NACL/tasks	Check AZ metrics
All targets unhealthy	Bad deep check	Adjust checks
Latency spike on failover	Cold connections	Pool refresh; rehearse
Retry amplification	No jitter/limit	Cap retries
Uneven AZ traffic	Misconfigured weights	Fix placement

Interview Questions and Answers

Q1: Why multi-AZ? A: Survive AZ failure domains without waiting for Region DR.
Q2: HA vs DR? A: In-Region continuity vs cross-Region recovery.
Q3: Health check tip? A: Match depth to risk; avoid self-DoS.
Q4: Idempotency? A: Safe retries for writes with keys.
Q5: Bulkhead? A: Isolate failure impact across resources.
Q6: Graceful degradation? A: Reduced UX when dependencies fail.
Q7: Timeout budget? A: Nested limits that fail fast coherently.
Q8: N+1? A: Capacity to lose one AZ and still serve.
Q9: Connection draining? A: Finish in-flight work on deregister.
Q10: Hidden SPOF examples? A: Single NAT, single Redis, single AZ ALB misconfig.
Q11: Retry storm? A: Coordinated client retries overwhelm survivors.
Q12: Cross-AZ cost? A: Data transfer charges—verify pricing.
Q13: Prove HA? A: Game day terminate AZ capacity; meet SLO.
Q14: Sticky sessions? A: Often accidental SPOF/scale limit—prefer stateless.
Q15: First five minutes AZ alarm? A: Confirm scope, watch ALB targets, avoid premature Region DR.

Hands-On Exercises

Exercise: Architecture review and resilience prioritization

Deliverable: SPOF list, ranked backlog, one improvement evidence or detailed design, alarms, ADR, cleanup.

Challenge ADR

Argue for multi-AZ DB cost against Finance using exam-day outage narrative and measured SLO cash impact (illustrative).

Lab evidence checklist

Budget; tags; before/after diagram; ranking; cleanup.

Narrative: grading backlog bulkhead

Certificate PDF workers saturate shared pools; lesson reads slow. Separate queues/pools restore reads; grading catches up.

Continuity

Next: FinOps, Cost Explorer and Cost Optimization—HA and DR spend must be intentional, tagged, and reviewed.

Deep dive: cellular architecture tease

Large platforms shard users into cells each multi-AZ. Blast radius shrinks. TechLearningPro may not need cells yet; know the pattern for interviews and future growth without implementing fantasy service meshes.

Deep dive: idempotency key storage

Store keys in DynamoDB/Redis with TTL; reject duplicates of quiz submits. Document key format (learnerId:quizId:attemptToken). Without this, retries double-write grades.

Deep dive: load shedding

Return 503 with Retry-After when overloaded; shed noncritical traffic first. Pair with ALB 5xx alarms. Shedding is reliability, not rudeness.

Extended enterprise scenario: live classroom AZ loss

During a virtual cohort, AZ-a tasks terminate. ALB shifts to AZ-b; brief elevated latency; degradation hides live reaction GIFs service; core quiz path lives. Status page notes “partial feature degradation.” DR not declared. This is HA success.

Extended enterprise scenario: cache as SPOF

Single-node cache failure stamps all requests through to DB; DB collapses. Multi-node cache or fail-open without cache for read-mostly catalogs saves the day. Document cache criticality honestly.

Extended practice scenario 1 — 46 high availability and fault tolerant architecture

Redraw a single-AZ TechLearningPro sketch into multi-AZ: subnets, ALB, ASG/ECS service, RDS Multi-AZ or Aurora. List remaining single points of failure.

In a timed drill for multi-AZ HA and graceful degradation, implement or design the core lab with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, architecture notes, and destroy/disable confirmation.

Discuss how this lesson’s tooling interacts with ELB/Route 53 (lesson 12), CloudWatch (lesson 36), DR (lesson 45), IAM least privilege, and CI promotion rules. List two failure modes you simulated, including detection and fix. Estimate cost drivers and teardown timing.

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps become flashcards. Premium courses finish when you can operate under time pressure without opening the console as a first resort.

Extended practice scenario 2 — 46 high availability and fault tolerant architecture

Define health check paths that prove dependency readiness (DB connectivity), not only process up.

Extended practice scenario 3 — 46 high availability and fault tolerant architecture

Tune load balancer unhealthy thresholds vs application startup time; document false flaps risk.

Extended practice scenario 4 — 46 high availability and fault tolerant architecture

Implement retry with exponential backoff and jitter on quiz submit client; show idempotency key design.

Extended practice scenario 5 — 46 high availability and fault tolerant architecture

Timeout budget: end-to-end 3s learner API—allocate ALB idle timeout, app timeout, DB timeout without exceeding.

Extended practice scenario 6 — 46 high availability and fault tolerant architecture

Bulkhead: isolate grading workers from lesson-read API thread pools / queues so grading backlog does not stall reads.

Extended practice scenario 7 — 46 high availability and fault tolerant architecture

Graceful degradation matrix: if recommendations ML service fails, serve static popular lessons; if forum fails, hide widget.

Extended practice scenario 8 — 46 high availability and fault tolerant architecture

Failure domain review: shared NAT, single Redis node, single Region Secrets Manager call patterns—rank blast radius.

Extended practice scenario 9 — 46 high availability and fault tolerant architecture

Chaos note: terminate one AZ’s tasks in staging; verify ALB drains; measure error rate vs SLO.

Extended practice scenario 10 — 46 high availability and fault tolerant architecture

Circuit breaker design for upstream payment partner; fallback messaging for purchases.

Extended practice scenario 11 — 46 high availability and fault tolerant architecture

Queue-based load leveling with SQS for certificate generation spikes (lesson 22 continuity).

Extended practice scenario 12 — 46 high availability and fault tolerant architecture

Sticky sessions anti-pattern hunt: find where TechLearningPro accidentally requires stickiness; eliminate or justify.

Extended practice scenario 13 — 46 high availability and fault tolerant architecture

Zonal DNS vs ALB: explain how AZ failure is absorbed without Route 53 failover (in-Region).

Extended practice scenario 14 — 46 high availability and fault tolerant architecture

Capacity planning: N+1 vs N+2 for ASG desired counts across AZs.

Extended practice scenario 15 — 46 high availability and fault tolerant architecture

Observability: alarms on AZ imbalance, target 5xx, queue depth—tie to lesson 36.

Extended practice scenario 16 — 46 high availability and fault tolerant architecture

Security interaction: security group referencing across AZs; avoid per-AZ manual drift.

Extended practice scenario 17 — 46 high availability and fault tolerant architecture

Cost of HA: duplicate AZ resources; compare to downtime cost story for Finance.

Extended practice scenario 18 — 46 high availability and fault tolerant architecture

Review ECS/EKS pod topology spread constraints conceptually for AZ spread—verify current scheduler features before claiming.

Extended practice scenario 19 — 46 high availability and fault tolerant architecture

RDS failover drill timing vs application reconnect behavior; connection pool invalidation.

Extended practice scenario 20 — 46 high availability and fault tolerant architecture

Write runbook: AZ impairment suspected—what to check before declaring Region DR (lesson 45).

Extended practice scenario 21 — 46 high availability and fault tolerant architecture

Graceful shutdown: SIGTERM handling for in-flight quiz posts; ALB deregistration delay alignment.

Extended practice scenario 22 — 46 high availability and fault tolerant architecture

Feature flag for degradation modes owned by product + platform jointly.

Extended practice scenario 23 — 46 high availability and fault tolerant architecture

Architecture review checklist of 15 SPOF questions for a peer’s diagram.

Extended practice scenario 24 — 46 high availability and fault tolerant architecture

Cleanup lab ASGs/ALBs created for chaos; budget remains.

Extended practice scenario 25 — 46 high availability and fault tolerant architecture

Full dress rehearsal: review architecture, prioritize top 5 resilience improvements with ROI narrative.

Compliance evidence pack for auditors

Architecture diagrams showing multi-AZ, game day results, SLO reports during AZ drills, exception register for single-AZ systems.

Glossary drill

Availability Zone; failure domain; health check; deregistration delay; idempotency; backoff; jitter; bulkhead; graceful degradation; load shedding; N+1; SPOF; Multi-AZ; connection draining.

Budget and cleanup sworn statement

I removed extra lab ASG capacity / ALBs / databases created for HA drills named ____. Budget alert remains enabled. Cost Explorer follow-up scheduled for cross-AZ and Multi-AZ charges.

Interview story to memorize

“We redesigned TechLearningPro progress API across three AZs with N+1 capacity, fixed deep health checks that were flapping, added jittered idempotent retries, and shipped a degradation mode for recommendations—AZ loss became a blip, not an outage.”

Module continuity paragraph

Lesson 45 recovers from disasters; lesson 46 prevents AZ blips from becoming disasters; lesson 47 asks whether the redundancy bill matches the risk. Reliability without FinOps is unfinished engineering.

Extended technical explanation: static stability

AWS Builders Library guidance on static stability encourages provisioning enough capacity that an AZ loss does not require scrambling for new EC2 capacity in the heat of an event. For TechLearningPro, that may mean running desired counts that already absorb AZ loss during peak exams, or using warm pools where documented for your compute mode. Auto Scaling that only reacts after CPU alarms may be too slow for abrupt AZ capacity cliffs. Measure scale-out duration in staging; if it exceeds error-budget burn tolerance, pre-provision. Document the chosen static-stability posture in the ADR so FinOps understands why “idle” capacity exists.

Extended technical explanation: multi-target observability during AZ events

Build a CloudWatch dashboard (or equivalent) with ALB HealthyHostCount per AZ, target 5xx per AZ, and application latency per AZ. Without per-AZ views, a brownout in one AZ looks like “the app is slow everywhere,” and engineers may incorrectly throttle global traffic. Correlate with RDS failover events and ECS task counts. Save a screenshot from a game day as evidence that the team can diagnose AZ scope in under five minutes—an interview-ready story and an operational necessity.

Extended design walkthrough: prioritizing the resilience backlog

Score each SPOF: learner impact (1–5), likelihood (1–5), fix cost (1–5 inverted), and implementation risk. Multiply for priority. Example: single-AZ RDS scores high impact/likelihood and may outrank polishing a tertiary microservice’s retries. Publish the scored backlog; resist executive pet projects that burn cycles on low-impact shine. Re-score quarterly as architecture evolves.

Hands-on narrative: intentional single-AZ failure in staging

In staging only, stop tasks in one AZ or remove the subnet from the service temporarily using documented APIs/console actions. Watch ALB host counts, error rates, and latency. Restore. Write three sentences on whether SLOs held. Never rehearse this first in production. Cleanup any temporary scaling changes and confirm desired counts return to baseline.

Budget and cleanup sworn statement

I verified the budget alert remains enabled. I destroyed or scaled down HA drill resources named ____. I confirmed no orphan NAT gateways, unused load balancers, or idle Multi-AZ databases remain from this lab unless intentionally retained for ongoing staging. I will re-check Cost Explorer within three days for EC2, ELB, RDS Multi-AZ, and cross-AZ data transfer line items. Signed ________ date ________.

Self-assessment rubric

Area	Weight
SPOF identification accuracy	25%
Prioritization rationale	20%
Concrete HA pattern application	25%
Observability for AZ scope	15%
Cleanup and cost honesty	15%

Final synthesis before summary

HA is topology, health, time budgets, isolation, and degradation—proven by drills, not diagrams alone. Prioritize SPOFs that hurt learners first.

Lesson Summary

Multi-AZ HA, honest health checks, disciplined retries/timeouts, bulkheads, and graceful degradation keep TechLearningPro available through everyday failures while DR handles rarer Region events. Review architectures for SPOFs, prioritize ruthlessly, instrument AZ-aware signals, budget duplicate capacity, and clean up labs. Tag Project=tlp-lesson46.

Next: FinOps, Cost Explorer and Cost Optimization.