Circuit Breaker

Introduction

Circuit breaker stops calling a failing dependency after error threshold, failing fast locally while the dependency recovers — then probes with half-open trials. Netflix Hystrix popularized the pattern for microservices; modern stacks use Resilience4j, Envoy outlier detection, and service-mesh outlier ejection.

Real production story

When Netflix's recommendation service slowed during a Cassandra hotspot, every homepage request blocked 30 threads waiting on timeouts. Thread pools exhausted across API tiers; unrelated profiles and search failed — classic cascading failure. Hystrix dashboards showed recommendation circuit closed but thread pool at 100% saturation from slow calls, not errors.

Engineers tuned circuit breakers on latency AND error rate: open when p99 > 2s or error > 50% over 10s window; half-open allows 5 probe calls. Fallback served trending defaults. Homepage availability recovered while recommendation team fixed Cassandra — blast radius contained to degraded UX, not total outage.

Business problem

Business pressure: Netflix homepage aggregates 20+ services — one slow dependency must not hold threads hostage for the entire streaming experience.

• Availability: Users tolerate generic rows; they do not tolerate spinner of death.
• Cost of cascade: Full outage during prime time dwarfs cost of fallback content.
• Ops clarity: Open circuit is actionable signal — "stop calling X" vs mystery timeouts.

Architecture overview

Closed: calls pass through. Open: fail immediately (fallback). Half-open: limited probes — success closes, failure reopens.

• Definition: Stateful wrapper around dependency calls with failure counting.
• Trip conditions: Error rate, slow call rate, or consecutive failures — sliding window.
• When to adopt: Every sync call to another team's service with user-facing latency SLO.
• Operability: Manual force-open for known incidents; metrics on state transitions.

Architecture motivation

Why architects care: Circuit breakers implement bulkhead at call level — fail fast beats fail slow when dependency is unhealthy.

• Force: Sync RPC chains amplify latency; threads are finite.
• Constraint: Fallback must be business-acceptable — not silent null pointers.
• Outcome: Per-dependency breaker state visible in dashboard; automated half-open recovery.

Internal architecture

Netflix Hystrix-style breaker per dependency:

HomepageAggregator
        │
        ├─ Breaker[recommendations] ──→ RecService (timeout 300ms)
        │       │ open? → TrendingFallback (Redis)
        │       └ half-open: 5 probes / 30s
        │
        ├─ Breaker[profiles] ──→ ProfileService
        └─ Breaker[continue-watching] ──→ CWService

State machine: CLOSED → OPEN (trip) → HALF_OPEN (timer) → CLOSED|OPEN
Trip: errorRate>50% OR slowCallRate>80% (10s window, min 20 calls)

Data flow

Breaker sits on outbound client — inbound APIs still need rate limits.

• Happy path: Call succeeds → reset failure counter → return result.
• Trip path: Threshold exceeded → open → immediate fallback without socket wait.
• Recovery: After sleep window, half-open probes → close on success streak.

System design diagram

Two diagrams show the Circuit Breaker topology and the primary request/event path used in production at scale.

Production code example

Resilience4j-style circuit breaker wrapper:

const breaker = new CircuitBreaker({
  failureRateThreshold: 50,
  slowCallRateThreshold: 80,
  slowCallDurationThreshold: Duration.ofMillis(300),
  slidingWindowSize: 20,
  waitDurationInOpenState: Duration.ofSeconds(30),
  permittedNumberOfCallsInHalfOpenState: 5,
});

async function getRecommendations(userId: string): Promise<Row[]> {
  return breaker.execute(async () => {
    const res = await recClient.fetch(userId, { timeoutMs: 300 });
    if (!res.ok) throw new DependencyError(res.status);
    return res.rows;
  }, async () => {
    metrics.breakerFallback.inc({ dependency: "recommendations" });
    return trendingCache.getForUser(userId);
  });
}

Enterprise case study

Netflix homepage recommendation circuit — contained Cassandra hotspot cascade.

• Before: Thread pool exhaustion; homepage hard down during rec service slowdown.
• Decision: Latency + error rate trip; trending fallback from Redis.
• After: Homepage UP with degraded rows; rec team fixed backend without company-wide incident.

Trade-offs

• Fallback quality vs availability: Stale trending rows beat outage — product must agree.
• False trips: Aggressive thresholds open during harmless blips — tune min sample size.
• Latency vs error tripping: Slow calls exhaust threads before error rate rises — monitor both.
• Hystrix deprecation: Move to mesh/outlier detection — concept remains essential.

Security considerations

Fallback data may be less fresh but must not bypass authZ.

• Cached fallbacks: Still scoped per user — no serve user A's continue-watching to user B.
• Force-open abuse: Admin API to force-open requires break-glass audit.
• DDoS interaction: Breaker helps shed load; pair with rate limiting at edge.

Scalability analysis

Each instance maintains breaker state — unsynchronized open across fleet is OK (fail fast everywhere).

• Horizontal scale: Per-process breakers; optional shared state via Redis for coordinated half-open.
• Thundering herd on close: Jitter sleep window when transitioning open → half-open.
• Cost: Fallback cache must scale to full traffic when breaker open — pre-warm trending sets.

Failure scenarios

Breaker open but fallback also fails — design layered degradation.

• Fallback cache miss: Serve static minimal response; never infinite internal retry.
• Flapping breaker: Sleep window too short — increase open duration exponentially.
• Hidden dependency: Breaker on A but A calls B — monitor entire chain or use mesh outlier.

Staff engineer insights

• A circuit breaker without fallback is just a faster error — product must define degraded mode.
• Trip on slow calls, not just 500 — thread exhaustion is the silent killer.
• Half-open probe count is a production knob — too many probes re-trip a recovering service.

Interview questions

Architecture review questions

• Trip conditions include latency/slow-call rate — not errors only.
• Fallback defined, tested, and scales when breaker open full traffic.
• Half-open probe limits configured — flapping monitored.
• Breaker metrics: state, trip count, fallback rate exported.
• Manual force-open runbook for dependency maintenance windows.
• Fallback responses enforce same authorization as live path.

Summary

Circuit breakers stop calling sick dependencies and route to fallbacks — containing blast radius like Netflix's homepage during recommendation outages. Trip on errors and latency; half-open probes recover safely.