Availability

Introduction

Availability is the proportion of time a system correctly performs its work — measured as nines against explicit SLOs, not uptime of individual VMs. Google SRE practice treats availability as an architecture attribute: redundancy, graceful degradation, and error budgets drive design for Search, Ads, and GCP control planes.

Real production story

A Google Cloud regional networking glitch dropped error budgets for a tier-0 internal admin API used by every SRE. The service had "high availability" on paper — three replicas — but all shared one Spanner instance and a synchronous dependency on a global LDAP sync. When LDAP slowed, thread pools exhausted and the API returned 503s globally. The fix was architectural: regional LDAP read replicas, bulkheads separating auth from data paths, and defined degraded mode (read-only admin) that preserves SLO during partial failure.

Business problem

Google products promise always-on access globally. Availability failures on Ads or Search have immediate revenue impact; internal platform outages multiply engineer toil across thousands of teams.

• Revenue SLO: Ads auction path targets 99.99%+ — minutes of downtime cost millions.
• Customer trust: GCP availability commitments are contractual — architectural gaps become SLA credits.
• Cascade risk: Shared platform outages create correlated failures across unrelated products.

Architecture overview

Availability = good requests / total requests over a window. Staff practice: define SLI, SLO, error budget; architect redundancy, isolation, and graceful degradation; verify with game days and fault injection.

• Failure domains: AZ, region, dependency, deployment — eliminate shared fate where possible.
• Graceful degradation: Shed load, serve stale, disable non-critical features — document each mode.
• Error budgets: Availability is a product decision — when budget burns, freeze risky launches.
• Toil vs reliability: Automate failover; manual runbooks do not scale to Google ops volume.

Architecture motivation

Availability requires design for failure: N+1 replicas alone do not create availability if dependencies, deployments, or data paths share fate. Google architects map failure domains and define degraded behavior.

• Force: Regional failures, dependency latency, and bad deploys are routine — not edge cases.
• Constraint: Strong consistency requirements limit naive active-active — choose per path.
• Outcome: Error budget policy gates feature velocity; architecture supports safe degradation.

Internal architecture

Google-style availability topology — eliminate single points across layers:

• Health checks must validate end-to-end capability — not just process up.
• Deploy strategies (canary, rollback) are availability tactics — one bad push is an outage.

Global anycast / GFE
   ↓
Regional clusters (independent failure domains)
   ├─ Service replicas (anti-affinity across zones)
   ├─ Bulkhead pools (critical vs batch)
   └─ Circuit breakers on dependencies
   ↓
Data layer
   ├─ Spanner / Bigtable (regional or global per RPO/RTO)
   └─ Read replicas + stale-while-revalidate
   ↓
Degraded mode controller (feature flags + capacity shedding)
   ↓
SLO dashboard + error budget policy

Data flow

Primary and fallback paths are first-class in architecture docs. Reads may failover to replica; writes may queue or reject with clear client semantics.

• Happy path: LB → healthy instance → primary store → response within SLO budget.
• Dependency slow: Timeout → circuit open → cached/stale response or fast fail with retry-after.
• Region loss: DNS/load shift to healthy region; RPO/RTO defined per data class.

System design diagram

Two diagrams show the Availability topology and the primary request/event path used in production at scale.

Production code example

Availability-aware HTTP handler — Go pattern with bulkhead and degraded mode:

• Degraded mode is explicit code path — not an incident-time hack.
• Bulkheads protect shared dependencies from exhausting thread pools.

func (h *AdminHandler) GetDashboard(w http.ResponseWriter, r *http.Request) {
  ctx, cancel := context.WithTimeout(r.Context(), 150*time.Millisecond)
  defer cancel()

  if h.degradedMode.Enabled() {
    h.serveCachedDashboard(w, r) // stale-while-revalidate, SLO-preserving
    return
  }

  err := h.authPool.Acquire(ctx) // bulkhead — max 50 concurrent auth calls
  if err != nil {
    http.Error(w, "service unavailable", http.StatusServiceUnavailable)
    return
  }
  defer h.authPool.Release()

  user, err := h.auth.Verify(ctx, r.Header.Get("Authorization"))
  if err != nil {
    metrics.AuthFailures.Inc()
    http.Error(w, "unauthorized", http.StatusUnauthorized)
    return
  }

  dash, err := h.repo.GetDashboard(ctx, user.ID)
  if err != nil {
    if cached, ok := h.cache.Get(user.ID); ok {
      h.writeJSON(w, cached, "X-Cache: stale")
      return
    }
    http.Error(w, "upstream error", http.StatusBadGateway)
    return
  }
  h.cache.Set(user.ID, dash)
  h.writeJSON(w, dash, "")
}

Enterprise case study

Google internal admin API hardening: Shared dependencies caused correlated unavailability despite multi-replica deploy.

• Before: 99.5% monthly availability; Sev-1 during every LDAP maintenance window.
• Decision: Regional read path, bulkheads, degraded read-only mode, quarterly game days.
• After: 99.95% SLO met; error budget funds feature work; MTTR under 15 minutes for dependency blips.

Trade-offs

• Availability vs consistency: CAP trade-offs explicit per endpoint — not one global choice.
• Cost vs nines: Each nine costs roughly 10× — 99.99% may suffice where 99.999% is over-engineering.
• Complexity vs redundancy: Multi-region active-active adds conflict resolution and ops burden.
• Feature richness vs degradation: More features mean more failure modes — tier critical paths.

Security considerations

Availability attacks: DDoS and resource exhaustion target availability — edge protection and rate limits are architectural.

• Edge absorption: WAF, CDN, and scrubbing before origin — do not expose raw service endpoints.
• Auth storms: Token validation overload during incidents — cache JWKS and short-circuit invalid tokens cheaply.
• Fail-open vs fail-closed: Security gates must degrade safely — document when to fail closed despite availability hit.

Scalability analysis

Availability at Google scale requires controlling blast radius as traffic grows — retries and fan-out can turn partial failures into total outages.

• Retry discipline: Exponential backoff with jitter; cap max retries — prevent retry DDoS on self.
• Load shedding: Prioritize tier-0 traffic when capacity constrained — architecture defines queues.
• Thundering herd: Cold start after regional failover — pre-warm and gradual traffic shift.

Failure scenarios

Availability incidents Google sees repeatedly: dependency cascade, overload after recovery, and "successful" health checks on broken services.

• Cascade: Auth service slow → all APIs 503 — bulkhead auth pools and token caching at edge.
• Recovery overload: Cache empty after restart — use stale-while-revalidate and request coalescing.
• Split brain: Misconfigured failover writes to two primaries — use quorum and fencing tokens.

Staff engineer insights

• Three replicas in one AZ is not high availability — draw failure domain boxes before counting nines.
• Error budgets connect availability architecture to product decisions — without them, SLOs are wallpaper.
• Graceful degradation must be tested; untested fallback paths fail during the incident when you need them most.

Interview questions

Architecture review questions

• Are SLI, SLO, and error budget defined and dashboarded?
• Are failure domains mapped with no hidden shared fate?
• Is graceful degradation documented and tested quarterly?
• Do deploy strategy and rollback meet availability targets?
• Are retries, timeouts, and circuit breakers configured on every dependency hop?
• Has a game day validated regional failover within RTO?

Summary

Availability at Google scale is an architectural discipline: failure domains, bulkheads, degraded modes, and error budgets — verified through SLO dashboards and game days, not replica counts alone.