Circuit Breaker

Introduction

Circuit Breaker stops cascading failures when a downstream dependency is unhealthy. After repeated failures the breaker opens — calls fail fast without waiting for timeout. After a cooldown it half-opens to test recovery. Combined with bulkheads and sensible retries, it keeps one slow payment API from taking down checkout.

The story

Partner payment API degraded; checkout retried aggressively, exhausting its thread pool and taking down the platform. Per-downstream breakers — open after 5 failures, half-open after 30s — limited blast radius. During a later 4-hour outage, checkout routed to backup provider while breaker stayed open.

The business problem

Slow or dead downstreams cascade through retry storms and resource exhaustion:

• Cascading failure: one slow API stalls all checkout threads.
• Retry storms: clients amplify outage hitting dead service harder.
• 30s timeouts: users wait instead of fast failover.
• Connection pool drain: every thread blocked on dead dependency.

The problem teams faced

Circuit Breaker fits when:

• Remote dependency can fail or degrade independently.
• Caller has fallback, cache, or degraded mode.
• Retry alone causes thread/connection exhaustion.
• You need fail-fast during known outages.

Understanding the topic

Intent: Trip open after failure threshold; fail fast; probe recovery in half-open.

• Closed — normal operation; failures counted.
• Open — fail immediately; no downstream call.
• Half-open — limited probe calls test recovery.
• Proxy/Decorator — wraps downstream client at seam.

Internal architecture

Breaker-wrapped payment client:

class PaymentClient {
  private state: "closed" | "open" | "half-open" = "closed"
  private failures = 0

  async charge(req: ChargeRequest): Promise<Receipt> {
    if (this.state === "open") throw new CircuitOpenError()
    try {
      const receipt = await this.stripe.charge(req)
      this.onSuccess()
      return receipt
    } catch (e) {
      this.onFailure()
      throw e
    }
  }
}

Visual explanation

Three diagrams cover breaker states, cascade prevention, and config:

Informative example

Before / after — retry storm to breaker + fallback:

// ❌ Retry until thread pool exhausted
async function pay(order) {
  for (let i = 0; i < 10; i++) {
    try { return await stripe.charge(order) }
    catch { await sleep(100) }  // hammers dead API
  }
}

// ✅ Breaker + fallback
async function pay(order) {
  if (breaker.isOpen("stripe")) return backupGateway.charge(order)
  try { return await breaker.exec("stripe", () => stripe.charge(order)) }
  catch { return backupGateway.charge(order) }
}

Execution workflow

Real-world use

Resilience4j, Polly (.NET), Istio outlier detection, Netflix legacy Hystrix patterns, AWS App Mesh.

Production case study

Payment API cascade incident:

• Symptom: partner API slow → retry storm → checkout platform down.
• Decision: breaker per payment provider + backup routing.
• Outcome: 4-hour partner outage; checkout stayed available.

Trade-offs

• Pro: limits blast radius; fail-fast UX; protects thread pools.
• Con: tuning required; false opens on transient blips.
• Con: fallback path must be tested — often neglected.

Decision framework

• Use on every critical remote dependency in microservices.
• Always pair open state with fallback or clear error.
• Don't retry non-idempotent calls blindly through closed breaker.

Best practices

• One breaker per downstream dependency — not global.
• Expose breaker state in metrics and health checks.
• Test fallback path in chaos drills.

Anti-patterns to avoid

• Breaker with no fallback — users get errors faster but not better.
• Same threshold for fast and slow dependencies.
• Infinite retry inside closed breaker.

Common mistakes

• Half-open allows too many probes — retriggers outage.
• Shared client without per-dependency breakers.

Debugging tips

• Graph breaker state transitions against downstream error rate.
• Trace: was call rejected by breaker or downstream?

Optimization strategies

• Adaptive thresholds from rolling error rate (Istio outlier detection).

Common misconceptions

• Circuit Breaker is Proxy/Decorator at resilience layer.
• Not a substitute for fixing downstream — contains damage.

Advanced interview questions

Summary

You can explain breaker states, prevent cascading failures, and pair with fallbacks. Tell the payment thread-pool story — if you can do that, you own Circuit Breaker.