Circuit Breaker, Retry and Bulkhead Patterns

Advanced System Design & Software Architecture · Lesson 41 · Microservices and Enterprise Architecture

Lesson Overview

Checkout calls payment through the Service name from lesson 39, with a connect timeout of 200 milliseconds and a response timeout of 800 milliseconds. It also asks a shipping quoter, a separate company, how long a printed workbook will take. On a quiet day both calls return. On the day the quoter stops answering, a naive client waits, retries three times, and occupies a connection for every learner who is still only trying to pay for a course that has no shipment. Payment was healthy. The saga still stalls in seats_reserved, because the thread or the connection pool that should have captured the card is full of shipping calls. That is a cascading failure. The timeouts were necessary and they were not sufficient.

This lesson puts four policies on those calls, using Resilience4j as it actually configures them, and it refuses the policies that make an outage worse. A timeout bounds one attempt. A retry spends more of the caller's budget on a fresh attempt, with backoff and jitter, and only when the attempt is safe to repeat. A bulkhead caps how many attempts may be in flight to that dependency, so shipping cannot consume the concurrency payment needs. A circuit breaker stops calling a dependency that is already failing, fails fast, and later lets a few probes through. Circuit Breaker Pattern, Retries & Exponential Backoff, Bulkhead Pattern, and Timeouts & Deadlines define the terms. Circuit Breaker, Retry Pattern, and Bulkhead do the same on the enterprise track. The decisions here are the ones those primers leave open: a card decline is not a failure of the breaker, a read timeout on capture is not a retry, and a fallback must not invent a successful payment.

The shopper-facing result stays the one lesson 32 and lesson 40 already named. Payment's open circuit becomes UNCERTAIN or a 503 with a problem type, never a captured receipt. Shipping's open circuit becomes a quoted estimate the product has agreed to show, or a hidden shipping panel. The edge rate limit in Rate Limiting, Throttling and Load Shedding protects TechLearningPro from its callers. The client-side limiter and the bulkhead protect payment and shipping from TechLearningPro.

Learning Objectives

By the end of this lesson you will be able to:

1. Set a per-attempt timeout that is shorter than the saga step and releases the HTTP connection, not only the caller's waiter.
2. Configure Resilience4j Retry, CircuitBreaker, and Bulkhead with the real builders and Spring Boot properties, including which exceptions count.
3. Explain the decorator order Retry around CircuitBreaker around Bulkhead, and what that order counts as a failure.
4. Keep retries off non-idempotent captures and off read timeouts whose outcome is unknown.
5. Isolate shipping concurrency from payment concurrency, including under virtual threads where the scarce resource is connections rather than platform threads.
6. Choose a fallback that cannot report money as captured, and a different fallback that may estimate shipping.
7. Trace a cascading failure from a slow quoter to a stuck saga, and name the metric that should have paged before the seats piled up.

Prerequisites

• Lesson 39's RestClient timeouts and the rule that readiness does not ping the card provider. The breaker is what reacts to the provider. Membership is not.
• Lesson 32's payment_uncertain state and the ban on releasing a seat while a capture might have succeeded. Retry policy that ignores this will double-sell.
• Lesson 37's idempotency key on the capture the provider honors. Idempotency and Exactly-Once Processing is why a connect failure may be retried and a guessed success may not.
• Lesson 40's status codes. A decline is a 409 with a problem type. Resilience logic must not map that to 200.
• Spring AOP, because the annotations do nothing on a self-call. resilience4j-spring-boot3 expects spring-boot-starter-aop and actuator.

Detailed Core Concepts

The budget comes first

The shopper's request has a budget, on the order of two and a half seconds for the pay click, set with the product. Payment's saga step gets at most one second of that, including retries. Shipping, when it is on the same click, gets at most four hundred milliseconds, and for a digital course it is not on the click at all. A configuration that sets three retries of an 800 millisecond payment timeout asks for 2.4 seconds of payment alone, before shipping, and multiplies load on the provider by three while it is sick. Write the budget down before you write maxAttempts. If the arithmetic does not fit, cut attempts or take shipping off the path. Do not raise the shopper's timeout to make a sample configuration fit.

Timeouts release the resource

SimpleClientHttpRequestFactory in lesson 39 sets connect and read timeouts on the client that owns the socket. That is the timeout that returns the connection to the pool. Resilience4j's TimeLimiter (TimeLimiterConfig.custom().timeoutDuration(...).cancelRunningFuture(true)) completes a CompletionStage with a timeout and tries to cancel the future. Cancellation does not reliably close a blocking socket that ignores interruption. A TimeLimiter around a blocking RestClient call can return control to the caller while the call still holds a connection until the read timeout fires. Use the client's timeouts as the source of truth for blocking I/O. Add TimeLimiter when the call is already asynchronous and cancellation propagates to the HTTP client you chose. The annotation @TimeLimiter applies to methods that return CompletionStage or CompletableFuture. Putting it on a method that returns CaptureReceipt directly is a misconfiguration, not a safer capture.

Two clocks still matter. The connect timeout should be short, 200 milliseconds inside one region, because a black-holed IP fails fast. The read timeout covers work the provider has accepted. They are not one number.

Retry is a second attempt, not a hope

Resilience4j RetryConfig sets maxAttempts including the first call, so maxAttempts(2) means one retry. The interval comes from IntervalFunction. Exponential backoff without jitter lines every caller up on the same sleep. IntervalFunction.ofExponentialRandomBackoff(Duration.ofMillis(50), 2.0, 0.5) spreads the wait. The YAML properties enableExponentialBackoff and exponentialBackoffMultiplier exist on resilience4j.retry.instances. The randomization factor for exponential backoff is applied in code through RetryConfigCustomizer, because the property file's enableRandomizedWait and randomizedWaitFactor are a separate switch and are easy to combine wrong. Prefer the explicit IntervalFunction for the shipping quoter so the jitter is visible in review.

What may be retried:

retryExceptions lists the types that retry. ignoreExceptions lists the types that must not, including CallNotPermittedException and the decline type. A configuration that retries Exception will retry declines, bugs, and open-circuit rejections. That is the retry storm with extra steps.

The circuit breaker counts dependency failures

CircuitBreakerConfig for payment:

javaCopy
CircuitBreakerConfig.custom()
    .slidingWindowType(CircuitBreaker.SlidingWindowType.COUNT_BASED)
    .slidingWindowSize(20)
    .minimumNumberOfCalls(10)
    .failureRateThreshold(50)
    .slowCallDurationThreshold(Duration.ofMillis(800))
    .slowCallRateThreshold(80)
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .permittedNumberOfCallsInHalfOpenState(5)
    .recordExceptions(ResourceAccessException.class, TimeoutException.class)
    .ignoreExceptions(PaymentDeclinedException.class)
    .build();

While the breaker is closed, calls pass through. Failures and slow calls, as defined by those thresholds, fill a sliding window. minimumNumberOfCalls stops a single failure at startup from opening the circuit. When the failure rate or the slow-call rate crosses the threshold, the breaker opens. Further calls throw CallNotPermittedException without touching the provider, until waitDurationInOpenState elapses. Then it is half-open and allows permittedNumberOfCallsInHalfOpenState trial calls. Enough successes close it. Enough failures open it again. DISABLED, FORCED_OPEN, and METRICS_ONLY exist for an incident: you can force the breaker open without pretending the thresholds did it, and you can record metrics without breaking calls while you tune the window.

A decline is ignoreExceptions. It is a business result. If declines count as failures, a promotion that produces many legitimate declines opens the circuit and blocks shoppers whose cards would have worked. Record timeouts, connection failures, and HTTP 5xx from the dependency. Do not record 400 from your own bad request.

Slow calls matter because a provider that returns in 5 seconds with HTTP 200 is still taking the fleet down. slowCallDurationThreshold matches the read timeout so "used the entire budget" counts as slow. A provider that is merely at 300 milliseconds is healthy relative to an 800 millisecond budget. Do not set the slow threshold to 50 milliseconds because a demo did. You will open the circuit on a normal p99.

The window is count-based here because payment's traffic is bursty during enrollment and a time-based window of ten seconds can be dominated by one quiet period or one spike. Revisit if the call rate is so low that twenty calls take an hour; minimumNumberOfCalls already prevents opening on a trickle, and a stale closed breaker is the failure mode at low traffic. Low traffic means a human should look before a robot opens.

Bulkheads are concurrency caps

BulkheadConfig.custom().maxConcurrentCalls(20).maxWaitDuration(Duration.ofMillis(20)).build() is a semaphore around payment calls. The twenty-first concurrent capture waits at most 20 milliseconds and then fails with BulkheadFullException. The caller maps that to "try later" or to UNCERTAIN according to whether a request was sent. If the bulkhead rejects before the HTTP call, nothing was sent, and the saga can retry the step later or fail the click as 503. It must not record a capture.

Shipping gets its own bulkhead, smaller, because quotes are optional on the pay click for digital goods and mandatory only on the workbook path. A full shipping bulkhead cannot fill payment's twenty permits. One shared bulkhead named downstream is how the quoter incident takes out capture. Name bulkheads after the dependency.

ThreadPoolBulkhead moves the call to another pool and returns a CompletionStage. It isolates CPU-blocking work from a small platform-thread server. It also drops the calling thread's ThreadLocal data unless you register a ContextPropagator, so the trace id and the security context disappear and you debug a capture with no tenant. Java 21 virtual threads, which Spring Boot 3.2 and later can use for the web server, remove the reason to hop pools just to get more threads. They do not remove the reason for a semaphore. The scarce resources are the provider's capacity, the HTTP connection pool, and the payment database pool. A virtual thread per request will happily open thousands of connections if the bulkhead is absent. Keep the semaphore. Skip the extra pool unless you have measured a blocking client that must stay off the request carrier.

Size the semaphore from the dependency's allowed concurrency and from your connection pool, not from the number of pods times a guess. If Hikari's pool for payment's own database is thirty, a bulkhead of two hundred on the capture path only queues inside the database pool and times out there instead of failing fast at the bulkhead. The bulkhead should be the smaller, earlier refusal.

Decorator order

The Resilience4j documentation composes decorators as Retry around CircuitBreaker around RateLimiter around TimeLimiter around Bulkhead around the function. The Spring aspects use that order unless you change retryAspectOrder and circuitBreakerAspectOrder. The consequence: each retry attempt is a separate call from the breaker's point of view, and the breaker can open in the middle of a retry sequence. The next attempt then sees CallNotPermittedException. Your retry config must ignore that exception so you do not spin the remaining attempts for nothing.

Putting the breaker outside a retry, by flipping aspect order, makes the breaker see one result after retries are exhausted. That hides a failing provider for longer and is the wrong default for payment. You want the breaker to open while the provider is timing out, and you want the number of attempts small so opening it does not first multiply traffic. Programmatic decoration makes the order obvious:

javaCopy
Supplier<CaptureReceipt> call = () -> http.capture(command);
Supplier<CaptureReceipt> isolated = Bulkhead.decorateSupplier(paymentBulkhead, call);
Supplier<CaptureReceipt> guarded = CircuitBreaker.decorateSupplier(paymentBreaker, isolated);
return Retry.decorateSupplier(paymentRetry, guarded).get();

The HTTP read timeout sits inside http.capture. There is no TimeLimiter in this stack. A client-side RateLimiter belongs here only if the provider contract says "no more than N calls per second from you." That is different from lesson 35, which limits shoppers at the edge. Do not use one configuration for both.

Fallbacks

A fallback runs when the decorators give up. For payment, the fallback returns CaptureReceipt with UNCERTAIN, or it throws a typed exception the saga already handles. It does not return CAPTURED. A cached last success, a stub "approved," or a queue-and-pretend-ok path will sell a seat without money. Shipping's fallback may return a documented estimate ("standard, price confirmed later") because the product accepted that lie of omission. Write the acceptance next to the method. If the product did not accept it, omit the quote and let checkout complete without a shipping line.

Annotation fallbacks must live on the same class, with the original arguments plus a final Throwable. If the method returns CompletableFuture<CaptureReceipt>, the fallback returns CompletableFuture<CaptureReceipt> too. A fallback that calls the same dependency again is a retry with the name fallback. Do not do it.

Spring annotations are AOP. A call from this.capture() inside the same bean skips the proxy, and the breaker never sees the call. The saga service calls a separate PaymentClient bean. A unit test that invokes the client through the Spring proxy, or that decorates programmatically, is the test that counts. A test that new's the class and calls the method will pass while production has no policy.

Architecture and Design Diagrams

mermaidCopy
flowchart TD
    saga[Checkout saga]
    retry[Retry payment: connect failure or documented 503 only]
    cb[Circuit breaker payment]
    bh[Bulkhead payment max 20]
    http[RestClient timeouts 200 ms and 800 ms]
    provider[Card provider]
    shipRetry[Retry shipping quote]
    shipCb[Circuit breaker shipping]
    shipBh[Bulkhead shipping]
    quoter[Shipping quoter]
    saga --> retry --> cb --> bh --> http --> provider
    saga --> shipRetry --> shipCb --> shipBh --> quoter

The two stacks share the saga and nothing else. A full shipping bulkhead does not reduce payment's twenty permits. The saga treats payment's CallNotPermittedException, bulkhead rejection before send, and read timeout as different inputs to the same state machine.

mermaidCopy
stateDiagram-v2
    [*] --> closed
    closed --> open: failure rate or slow-call rate over threshold
    open --> halfOpen: waitDurationInOpenState elapsed
    halfOpen --> closed: trial calls succeed
    halfOpen --> open: trial calls fail
    closed --> closed: decline ignored

Declines leave the breaker closed. They still produce a compensated saga. Opening is reserved for "the dependency is not answering within the budget."

Technical Explanation

What a storm looks like in the numbers

Fifty checkout pods, each willing to run twenty concurrent captures, each retrying twice, is two thousand in-flight attempts if everything aligns. The provider's limit is a few hundred. The rest sit in timeouts. Each timeout holds a connection for 800 milliseconds, then the retry adds another. Meanwhile shipping, if it shares the pool, holds connections for its own longer timeout. The EndpointSlice from lesson 39 still lists every pod as ready, because readiness does not call the provider. That is correct and it will confuse an operator who thinks "the pods are healthy, so the client is wrong." The pods are healthy. The client policy is amplifying. The breaker is what cuts the amplification once the failure rate in the window crosses fifty percent. Until the window fills, the bulkhead is what caps each pod at twenty payment calls and a separate, smaller number of shipping calls. Without the bulkhead, the breaker opens late, after the pools are already exhausted.

Jitter matters in that window. If every pod sleeps exactly 50 milliseconds then exactly 100, the provider sees gaps and then cliffs. ofExponentialRandomBackoff with a factor of 0.5 spreads the second wave. It does not reduce the total number of attempts. maxAttempts does that. Jitter without a cap is politeness. The cap is the control.

Mapping outcomes to the saga

The dangerous row is the read timeout, and it stays dangerous if a retry policy treats ResourceAccessException as always retryable. Spring's ResourceAccessException wraps both connect failures and read timeouts. Split them. Retry ConnectException. Do not retry SocketTimeoutException on capture. A single retryExceptions: ResourceAccessException entry is the bug that looks like a one-line configuration improvement.

Shipping can leave the pay click

For a digital course, do not call the quoter during pay. The dependency that is allowed to fail the click is payment. Workbook checkout may call the quoter under its bulkhead, and on BulkheadFullException or an open circuit it uses the estimate fallback or omits the delivery date. It does not hold the seat reservation open while the quoter is down. Seat leases are about payment uncertainty, not about printing schedules.

Metrics that page

resilience4j-micrometer registers meters the actuator can scrape. Alert on the payment breaker's open state, on BulkheadFullException rate, and on the age of sagas stuck in seats_reserved and payment_uncertain from lesson 32. A rising retry count while the breaker is closed is the window before the trip; it is worth a warning, not a second retry layer. Do not alert on declines. You will train the on-call to ignore the payment page.

Force-open the payment breaker in a game day and confirm checkout returns the problem type from lesson 40, seats follow the uncertain-or-not-sent rule above, and shipping quotes still succeed. If shipping dies in that test, the bulkheads are shared or the client is shared with one connection pool of size ten. Give the shipping RestClient its own connection pool and its own breaker name.

Annotation shape, when you use it

javaCopy
@CircuitBreaker(name = "payment", fallbackMethod = "captureFallback")
@Retry(name = "payment")
public CaptureReceipt capture(CaptureCommand command) {
    return http.capture(command);
}

private CaptureReceipt captureFallback(CaptureCommand command, Throwable failure) {
    if (failure instanceof PaymentDeclinedException declined) {
        throw declined;
    }
    return new CaptureReceipt(null, CaptureStatus.UNCERTAIN);
}

The fallback that rethrows the decline matters. Some examples swallow every Throwable and return a default. That default would turn a decline into UNCERTAIN and hold the seat until reconciliation, which is safer than reporting success and worse than compensating immediately. Rethrow known business exceptions. Map only dependency failures to UNCERTAIN.

YAML for the shipping retry's exponential backoff, with jitter applied in a customizer:

yamlCopy
resilience4j:
  retry:
    instances:
      shippingQuote:
        maxAttempts: 3
        waitDuration: 50ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - org.springframework.web.client.ResourceAccessException
  circuitbreaker:
    instances:
      payment:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        minimumNumberOfCalls: 10
        failureRateThreshold: 50
        slowCallRateThreshold: 80
        slowCallDurationThreshold: 800ms
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 20
        maxWaitDuration: 20ms
      shippingQuote:
        maxConcurrentCalls: 8
        maxWaitDuration: 20ms

javaCopy
@Bean
RetryConfigCustomizer shippingJitter() {
    return RetryConfigCustomizer.of("shippingQuote", builder ->
        builder.intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
            Duration.ofMillis(50), 2.0, 0.5)));
}

The customizer overrides the interval. The exception lists stay in YAML where operators can see them. Do not also set enableRandomizedWait beside this customizer and hope the factors compose in the way you remember. One place owns the wait.

Half-open traffic is real traffic. Five permitted calls are five captures or five quotes, not synthetic pings, unless you add a health endpoint that does not move money. Prefer letting a real quote through for shipping. For payment, five trial captures must still carry idempotency keys and go through the saga. A half-open probe that uses a fake order id will either fail in a new way or, worse, capture a real card if someone points it at production "just to test the breaker."

Practical Implementation or Design Walkthrough

Problem

Payment and the workbook shipping quoter are both called from checkout. The provider sometimes refuses the connection, sometimes captures and loses the response, and sometimes declines. The quoter sometimes stalls for thirty seconds. Digital purchases must still complete when the quoter is down.

Requirements

• One capture attempt's read timeout is 800 milliseconds. At most one retry, and only when the provider cannot have seen the request, or when the provider's idempotency contract covers a 503.
• A read timeout stores payment_uncertain and does not retry inside the client.
• Declines do not open the breaker and do not retry.
• Shipping in flight cannot reduce payment's concurrency cap.
• A payment fallback never returns CAPTURED.
• Digital pay does not call shipping.

Steps

1. Split exception types in the HTTP adapter: connect failure, read timeout, HTTP 5xx, decline.
2. Build the payment stack programmatically in the order bulkhead, then breaker, then retry, with the client's timeouts inside the function. Ignore CallNotPermittedException and declines on the retry. Ignore declines on the breaker.
3. Give shipping its own breaker, bulkhead, RestClient, and connection pool. Retry the quote with ofExponentialRandomBackoff. Fallback to the estimate only on the workbook path.
4. Remove the shipping call from the digital pay path. Confirm with a trace assertion in a test.
5. Map outcomes to saga transitions in one table in the code review, the table above, and test each row.
6. Scrape the breaker state and the bulkhead rejection count. Add the alert. Run a game day that stalls the quoter and declines a card in the same minute.

Failure handling

Quoter stalled: shipping bulkhead fills, further quotes fail in 20 milliseconds, payment's twenty permits stay available, digital pay captures. Provider read timeouts: no client retry, saga payment_uncertain, reconcile. Provider connection resets: one retry, then unavailable. Declines: compensate, breaker stays closed. Open payment breaker: shopper sees the problem type for dependency unavailable, no second capture, reservation behavior follows whether a request was sent.

Validation

• A test decline storm of one hundred calls leaves the breaker closed.
• A test of eleven timeouts in a window of twenty opens it, and the next call does not increment the HTTP server's request count.
• A read timeout produces one HTTP request, not two.
• A connect failure produces at most two.
• Filling the shipping bulkhead does not change payment bulkhead's available permits.
• Self-invocation test: a same-class call is either forbidden by review or covered by a proxy test that would fail if the annotation were the only protection. Prefer the programmatic stack so AOP is not required for correctness.
• Fallback unit test: every dependency exception type returns UNCERTAIN or throws, and none return CAPTURED.

Enterprise Real-World Scenarios

Scenario 1: Workbook quoter stalls during enrollment

Context. A cohort buys a course plus a printed workbook. The quoter's p99 goes to thirty seconds. Checkout used one RestClient, one pool, and retryExceptions of Exception with three attempts. Seats stay reserved until the lease sweeper runs, which is longer than the pay budget. Payment's provider dashboard shows little extra error; the calls never got out.

Design. The split stacks above. Digital orders skip the quoter. Workbook orders take the estimate fallback when the shipping breaker is open, and the saga does not wait on an estimate to capture. The seat lease is not extended for shipping. Operators get a shipping-breaker page and a payment-breaker page that stayed quiet, which is the success condition.

What you refuse. Raising the shared pool to five hundred connections. That moves the stall into the provider and the database, and it looks like a fix until enrollment's second hour.

Scenario 2: Declines open the circuit on a bad promotion

Context. A coupon campaign produces a wave of legitimate issuer declines. The breaker recorded every 4xx. It opened. Shoppers with valid cards received 503 for thirty seconds, retried, and a few of those retries double-submitted because the client treated 503 as a new order without the original idempotency key.

Design. ignoreExceptions includes the decline type. The campaign's declines show up as a business metric, not as a dependency failure. 503 from an open breaker, when it does happen, is retried by the partner with the same Idempotency-Key from lesson 40. The breaker opening is no longer the path that handles a pricing mistake. Product and provider errors stop sharing one counter.

Best Practices

• Write the time budget, then set timeouts and maxAttempts so the product of attempts and timeouts fits inside the saga step.
• Let the HTTP client's connect and read timeouts free the connection. Do not wrap a blocking client in TimeLimiter and call that a timeout.
• Retry reads and connect failures. Do not retry a capture read timeout. Split ResourceAccessException before you match on it.
• Require the provider's idempotency contract before you retry any capture that might have arrived.
• Ignore declines and CallNotPermittedException on the retry. Ignore declines on the breaker.
• Keep minimumNumberOfCalls high enough that one error does not open the circuit, and the slow-call threshold near the real read timeout.
• One bulkhead and one connection pool per dependency. Size the bulkhead inside the pool size.
• Prefer a semaphore bulkhead. Add a thread-pool bulkhead only when you must move blocking work and you have a ContextPropagator.
• Fall back to UNCERTAIN or a typed error for payment. Never fall back to CAPTURED.
• Call the decorated bean through the proxy, or decorate in code so a self-call cannot skip the policy.
• Alert on open breakers and bulkhead rejections. Do not alert on declines.
• Half-open trials are real calls. They use real idempotency keys.
• Keep lesson 35's edge limiter and this client limiter as separate policies with separate numbers.
• Game-day the quoter stall and the decline storm before enrollment week.

Common Mistakes and Anti-Patterns

Retrying Exception. Declines, bugs, and bulkhead rejections all retry. The provider sees a storm of cards it has already declined, and the breaker opens for the wrong reason, or the bulkhead stays full of retries. List the exceptions.

One timeout number. Connect and read are different failures. A thirty-second read timeout copied from a sample exceeds the saga budget and holds every permit in the bulkhead. The bulkhead then rejects healthy work.

Shared bulkhead named default. Shipping and payment share a semaphore. The first dependency to stall owns the process. The diagram in the design review should show two names, and the test should fill one and use the other.

Fallback returns success. The stub receipt marks the saga completed. Finance finds captures that the provider never saw, or misses captures the stub invented. Payment's fallback is uncertainty or an error. There is no third option that says the money moved.

Breaker records declines. A commercial event looks like an outage. Valid cards fail for waitDurationInOpenState. Filter at the exception type, and check the filter with a test that would fail if someone "simplifies" recordExceptions to Exception.

Self-invocation. The annotation is on the class that calls itself. Production has no breaker. The code review rule is: the saga depends on an interface, and the implementation is the decorated client.

TimeLimiter as a substitute for a socket timeout. The caller unblocks. The connection stays checked out. The pool exhausts on schedule, and the breaker sees timeouts from the pool rather than from the provider. Fix the client timeout first.

Retrying the open circuit. CallNotPermittedException is in retryExceptions by way of a broad superclass. Each shopper burns attempts that cannot succeed and adds latency inside a failure you already understood. Ignore that exception and return.

Troubleshooting and Design Review

Design review: What is the shopper budget, and what is the arithmetic of attempts times timeouts? Which exceptions retry, and which of those might mean the provider already captured? Which exceptions move the breaker, and is a decline among them? If shipping stalls, what is the maximum number of in-flight payment calls? What does the fallback return, and can that value be CAPTURED? Does readiness stay up when the breaker is open? Which metric pages a human?

Interview Questions and Answers

Q1. What is the difference between a timeout, a retry, and a circuit breaker on the payment call?

A timeout ends one attempt so the connection and the saga step cannot wait forever. A retry spends another attempt, only when repeating the call is safe. A circuit breaker watches the outcomes and, once the failure or slow-call rate in the window is too high, stops calling until a trial succeeds. Timeouts without a breaker let every shopper pay the full timeout cost for the whole incident. Retries without a cap and without jitter multiply that incident. The breaker is the back-pressure on TechLearningPro's own clients. Timeouts & Deadlines and Circuit Breaker define each knob. The payment policy uses all three, with different exception lists.

Q2. Why is a card decline not a circuit-breaker failure?

A decline means the provider answered and the business result is no. Recording it moves failureRateThreshold for a reason that another shopper's valid card does not share. The breaker opens, and lesson 40's 503 replaces a normal compensate path. Put the decline type in ignoreExceptions on both the breaker and the retry. Record connection failures, read timeouts, and provider 5xx. Watch declines on a business dashboard, not on the outage page.

Q3. When will you retry a capture, and when will you reconcile instead?

Retry a connect failure, where the provider could not have seen the body, once. Retry a 503 only if that provider documents that the idempotency key makes the retry return the original result. Do not retry a read timeout. Spring wraps both connect and read problems in ResourceAccessException, so matching the wrapper retries the timeout. A timeout stores payment_uncertain and the saga queries by key, which is lesson 32. Retrying it is how you double-charge a provider that did capture and lost the bytes on the way back.

Q4. Why does Resilience4j's default order put Retry outside the CircuitBreaker, and what do you configure because of that?

Each attempt is recorded separately, so a sick provider trips the breaker sooner, and a retry in progress can observe CallNotPermittedException once the breaker opens. Ignore that exception on the retry so an open circuit fails the saga immediately instead of burning the rest of maxAttempts on calls that will not leave the process. Flipping the order so the breaker sees only the final retry result delays the trip and is a poorer fit for payment. Decorate in code as bulkhead, then breaker, then retry, matching the documented order, and the review can see it.

Q5. What does a bulkhead still do when the web server uses virtual threads?

It caps in-flight calls to one dependency. Virtual threads make it cheap to block a carrier; they do not add provider capacity, HTTP connections, or database connections. Without the semaphore, a stall becomes thousands of blocked calls and a full Hikari pool. maxConcurrentCalls should be lower than that pool. maxWaitDuration should be short so the extra callers fail instead of waiting out the timeout. A separate bulkhead for shipping is what keeps a quoter stall from occupying payment's permits. A ThreadPoolBulkhead is optional and drops ThreadLocal context unless a ContextPropagator is registered. The semaphore is the default.

Q6. What may a fallback return for payment, and what may it return for a shipping quote?

Payment's fallback returns UNCERTAIN or throws a typed dependency error. It must not return CAPTURED, including from a cache or a stub. A decline is rethrown so the saga compensates. Shipping's fallback may return a product-approved estimate or omit the quote, because no money moves on that response. A fallback that calls the same dependency again is an unlabeled retry. Same-class annotation fallbacks need the original arguments plus Throwable, and a CompletionStage method needs a CompletionStage fallback.

Q7. How do you stop a retry storm from making a provider outage worse?

Fit maxAttempts times the read timeout inside the saga budget, so payment is two attempts at most and often one. Add jitter with IntervalFunction.ofExponentialRandomBackoff so pods do not retry in lockstep. Open the breaker on the failure rate so later calls do not reach the provider. Cap concurrency with the bulkhead. Do not retry when the breaker is open. Edge shedding from lesson 35 reduces how many shoppers enter. None of these by itself is enough; a breaker without a bulkhead still allows a full window of concurrent timeouts before it opens.

Q8. Readiness stays green while the breaker is open. Is that a bug?

No. Lesson 39 made readiness mean "this process and its database can accept work." The provider being down is why the breaker opened. Removing the pod from the Service would leave checkout with nobody to fail fast and nobody to reconcile. The shopper-facing 503 comes from the client policy, not from an empty EndpointSlice. An operator who restarts pods to "fix" an open breaker is treating a dependency incident as a process incident and will drop in-flight reconciliations.

Q9. A TimeLimiter of 800 milliseconds wraps a blocking RestClient whose read timeout is 30 seconds. What happens under load?

The limiter completes the caller's future at 800 milliseconds and attempts to cancel. The blocking read often continues until 30 seconds and holds the connection. Enough of those calls exhaust the pool. New calls time out on pool checkout, the breaker records those as failures, and you page on a pool problem you created. Set the client's read timeout to 800 milliseconds and let that be the failure the breaker records. Use TimeLimiter when cancellation actually aborts the downstream call.

Q10. How does this interact with the public order API and the saga?

The API never turns a breaker rejection into HTTP 200. It uses the problem type for dependency failure from lesson 40, or it returns the order in payment_uncertain if a request might have reached the provider. Partners retry with the same idempotency key. The saga's compensate path runs for declines only. Half-open trial calls are real captures through that saga, not a side door. Contract tests do not prove any of this. The tests that fill one bulkhead and decline a hundred payments are the ones that do.

Q11. Where do you put a client-side rate limiter relative to lesson 35?

Lesson 35 limits inbound shoppers at the gateway so one tenant cannot flood checkout. A Resilience4j RateLimiter on the shipping or payment client limits what checkout sends to that provider, matching a contractual ceiling. The numbers differ, the windows differ, and a single shared limiter will either throttle shoppers because the provider is slow or throttle the provider because a shopper is busy. Compose the provider limiter inside the same stack as the breaker if you need it, and do not count a rate-limited local rejection as a provider failure.

Hands-On Exercises

Exercise 1: Two dependencies, one click

Problem. The pay click may capture a card and, for workbooks, quote shipping. Digital pay must ignore the quoter.

Tasks.

1. Write the budget arithmetic for payment and for shipping.
2. Specify retry exceptions for each call, including the read-timeout case.
3. Configure two bulkheads and say what happens when the shipping permits are all taken.
4. Define the payment fallback's return value and the test that forbids CAPTURED.

Deliverables. The tables, the Resilience4j configuration, and the saga mapping for timeout versus decline versus bulkhead rejection.

Exercise 2: Decline storm and quoter stall

Problem. In one game day the issuer declines a large share of cards, and the quoter stops responding.

Tasks.

1. Predict the breaker state for payment and for shipping after two minutes, given ignoreExceptions for declines.
2. Say which alert fires, and which alert firing would mean the exception filter is wrong.
3. Explain the shopper's HTTP result and the seat's state for a digital purchase during the stall.

Rubric.

Lesson Summary

Payment and shipping get separate timeouts, retries, breakers, and bulkheads. The HTTP client timeout is what frees the connection. Retries cover shipping reads and payment connect failures, not capture read timeouts and not declines. The breaker opens on dependency failure and slow calls, stays closed across a decline campaign, and fails fast with CallNotPermittedException, which the retry must not spin on. The bulkhead caps concurrency even when virtual threads make blocking cheap, and shipping's cap cannot eat payment's. The only acceptable payment fallback is uncertainty or a typed error. The saga already knows what to do with that uncertainty: reconcile, and do not release a seat while a charge might exist.

An open breaker is not a reason to take the pod out of the Service. It is a reason the shopper sees a problem document and the on-call sees a metric. The next lesson changes the shape of the data behind a different kind of read/write split. CQRS and Event Sourcing is for the ledger and the projections, not a replacement for the breaker you put around the provider.

Further Reading

• Resilience4j documentation: circuit breaker, retry, bulkhead, and time limiter, including decorator order and IntervalFunction.ofExponentialRandomBackoff.
• resilience4j-spring-boot3 configuration properties: slidingWindowSize, minimumNumberOfCalls, failureRateThreshold, slowCallDurationThreshold, waitDurationInOpenState, maxConcurrentCalls, enableExponentialBackoff.
• RetryConfigCustomizer for an interval function the YAML does not express cleanly.
• Spring Framework SimpleClientHttpRequestFactory connect and read timeouts.
• Michael Nygard, Release It!, on circuit breakers, bulkheads, and cascading failure.
• Marc Brooker, "Exponential Backoff And Jitter," Amazon Builders' Library.
• This course: Distributed Transactions and Saga Pattern, Service Discovery and Configuration Management, API Design, Versioning and Contracts, Rate Limiting, Throttling and Load Shedding, Idempotency and Exactly-Once Processing.
• Primers: Circuit Breaker Pattern, Retries & Exponential Backoff, Bulkhead Pattern, Timeouts & Deadlines, Circuit Breaker, Retry Pattern, Bulkhead.