Retry Pattern

Introduction

Retry pattern re-invokes failed operations with controlled backoff and jitter when failures are transient — timeouts, 503s, partition leader elections. Google gRPC clients and Cloud Tasks embed exponential backoff by default; unbounded retries cause retry storms that rival the original outage.

Real production story

Google Cloud Spanner client teams observed cascading latency when a regional backend returned 503 for 90 seconds. Thousands of microservices retried synchronously every 100ms with no jitter — Spanner frontend CPU saturated on retry traffic alone, extending the outage from 90s to 14 minutes. SRE postmortem labeled it "metastable failure."

Platform libraries shipped retry policies: max attempts, exponential backoff with full jitter, retry only on idempotent ops and defined gRPC status codes. Services opted in via config; default max backoff 32s. Retry-induced amplification dropped 85% in the next similar incident.

Business problem

Business pressure: Google services depend on hundreds of RPC dependencies — transient failures are normal; giving up on first timeout loses revenue and breaks SLAs.

• Reliability: Without retries, 0.1% blip becomes user-visible error.
• Stability: With bad retries, 0.1% blip becomes regional outage.
• Developer velocity: Central retry libraries prevent each team inventing dangerous loops.

Architecture overview

Exponential backoff: delay = min(cap, base × 2^attempt). Full jitter: sleep = random(0, delay) — spreads retry thundering herd. Retry budget: max attempts + total timeout aligned with client deadline.

• Definition: Transient failure → wait → retry until success or policy exhausted.
• When to retry: UNAVAILABLE, DEADLINE_EXCEEDED (careful), connection reset — not INVALID_ARGUMENT.
• When not to: 400-class errors, business rule violations, non-idempotent POST without key.
• Operability: Metric retry_count per dependency; alert on retry ratio spikes.

Architecture motivation

Why architects care: Retries are load multipliers — architecture must cap amplification and require idempotency.

• Force: Partial failures dominate at scale; success path is insufficient design.
• Constraint: Retries only safe on idempotent operations or with idempotency keys.
• Outcome: Standard RetryPolicy in gRPC/HTTP clients; budget aligned with deadline.

Internal architecture

Google gRPC retry interceptor stack:

ClientStub (deadline: 3s total)
        ↓
RetryInterceptor
  policy: maxAttempts=5
          backoff: initial=100ms, multiplier=2, max=2s
          jitter: FULL
          retryableCodes: [UNAVAILABLE, ABORTED]
        ↓
CircuitBreakerInterceptor (optional)
        ↓
HTTP/2 → BackendService

❌ while(true) retry without backoff
✅ total retry time < client deadline
✅ idempotency token on mutating RPCs

Data flow

Retries live in the client (or sidecar) — not scattered in business code.

• Sync path: Interceptor catches retryable error → sleep jitter → re-issue RPC with same idempotency key.
• Async path: Cloud Tasks / Pub/Sub push with maxDeliveryAttempts + minBackoff.
• Give up: Return error to caller; trigger circuit breaker after threshold.

System design diagram

Two diagrams show the Retry Pattern topology and the primary request/event path used in production at scale.

Production code example

Exponential backoff with full jitter — production retry helper:

const RETRYABLE = new Set(["UNAVAILABLE", "DEADLINE_EXCEEDED", "RESOURCE_EXHAUSTED"]);

async function callWithRetry<T>(
  fn: () => Promise<T>,
  opts: { maxAttempts: number; baseMs: number; capMs: number; deadlineMs: number },
): Promise<T> {
  const start = Date.now();
  let attempt = 0;
  while (true) {
    try {
      return await fn();
    } catch (err) {
      attempt++;
      if (!RETRYABLE.has(err.code) || attempt >= opts.maxAttempts) throw err;
      if (Date.now() - start >= opts.deadlineMs) throw err;
      const exp = Math.min(opts.capMs, opts.baseMs * 2 ** (attempt - 1));
      const jitter = Math.random() * exp;
      await sleep(jitter);
    }
  }
}

Enterprise case study

Google Spanner client retry policy rollout after metastable retry storm incident.

• Before: 90s backend blip extended to 14 min regional degradation.
• Decision: Platform RetryPolicy with jitter; retry only idempotent + defined codes.
• After: Retry amplification down 85%; libraries default-safe for new services.

Trade-offs

• Latency vs success rate: More retries improve success but blow p99 — cap by deadline.
• Jitter vs predictability: Full jitter spreads load but harder to debug timing.
• Central library vs custom: Platform policy ensures safety; edge cases need override with review.
• Idempotency requirement: Retries on non-idempotent ops cause duplicates — fix architecture first.

Security considerations

Retries amplify auth failures and brute force if misconfigured.

• Do not retry 401/403: Credential rotation needed, not backoff.
• Rate limits: Retrying 429 without honoring Retry-After worsens ban.
• Idempotency keys: Same key on every retry attempt for mutating calls.

Scalability analysis

Retry storms scale with caller count × retry rate — metastable failures are Google SRE textbook material.

• Horizontal scale: Jitter essential when 10k pods retry same dependency.
• Retry budget: Service mesh may cap retries per second to protect backends.
• Cost: Billed RPCs multiply with retries — track effective QPS vs raw QPS.

Failure scenarios

Retry without idempotency and without jitter are the top production footguns.

• Retry storm: Backend at 50% capacity; retries push to 0% — break circuit, shed load.
• Deadline exceeded loop: Retry after deadline wasted — respect context cancellation.
• Poison 500: Bug returns 500 always — max attempts then DLQ, not infinite loop.

Staff engineer insights

• If your retry policy has no max attempts, you have an infinite loop with extra steps.
• Full jitter is not optional at Google scale — synchronized backoff is a distributed bug.
• Measure retry ratio per dependency — sudden 10× means upstream is hurting, not "network flakiness".

Interview questions

Architecture review questions

• Max attempts and total deadline defined — no unbounded loops.
• Full jitter on backoff — verified in client library config.
• Retry only on idempotent ops or with idempotency keys.
• Non-retryable status codes enumerated (4xx business errors).
• retry_count metric exported per downstream dependency.
• Runbook for retry storm — disable retries via feature flag.

Summary

The retry pattern handles transient distributed failures with exponential backoff and jitter. Google's production experience proves unbounded synchronous retries are dangerous — cap attempts, require idempotency, and integrate with circuit breakers for enterprise resilience.