Resilience Patterns

Cascading failure follows a predictable path: a dependency slows → callers block waiting → their thread pools fill → health checks fail → load balancers remove healthy instances → remaining instances take more load → the system enters a death spiral. Netflix coined much of the modern vocabulary (timeouts, circuit breakers, bulkheads) after learning this the hard way on AWS.

Resilience patterns do not prevent failures—they bound them. You decide how long to wait, how many times to retry, when to stop calling a sick dependency entirely, and what degraded experience to show users instead of a blank 500 page. These policies belong in client libraries, service mesh, and API gateway layers; domain code should declare intent, not rediscover TCP semantics.

flowchart LR
  U[Users] --> A[Checkout]
  A --> B[Payment]
  A --> C[Inventory]
  A --> D[Recommendations]
  D -. slow .-> A
  A -. threads blocked .-> U

Timeouts cap how long a caller waits for a response. They convert “hang indefinitely” into a controlled failure you can retry, circuit-break, or fallback. The hard part is choosing values: too aggressive and you fail healthy services under normal tail latency; too lenient and you absorb cascading delay.

Setting timeouts in practice

• Start from SLO p99 latency of the dependency plus network margin—not from “30 seconds feels safe.”
• Align end-to-end user timeout > sum of critical path service timeouts, or users see errors while backends still work.
• Use separate connect vs read timeouts: DNS/TLS stalls differ from slow business logic.
• Propagate deadlines with gRPC deadline context or HTTP Timeout headers where supported.

resilience4j.timelimiter:
  instances:
    inventoryClient:
      timeoutDuration: 2s
      cancelRunningFuture: true

feign:
  client:
    config:
      inventoryClient:
        connectTimeout: 500
        readTimeout: 2000

Retries re-issue failed calls when failure looks temporary: connection reset, HTTP 503, gRPC UNAVAILABLE. Blind retries on every error amplify load during outages—the classic retry storm that keeps a recovering service down.

Exponential backoff with jitter

Wait time grows exponentially between attempts: 100 ms → 200 ms → 400 ms, capped at a max. Jitter randomizes delay within a range so thousands of clients do not retry in sync (thundering herd). Formula (full jitter): sleep = random(0, min(cap, base * 2^attempt)).

Idempotency requirement

A retry-safe operation produces the same effect whether executed once or five times. GET and PUT are naturally idempotent; POST creating resources needs an Idempotency-Key header stored server-side. Payment captures and inventory reservations must deduplicate on business key, not HTTP method alone.

What NOT to retry

• 4xx client errors (400, 404, 422)—repeating will not fix bad input.
• 429 Too Many Requests—retry only if you honor Retry-After and backoff aggressively.
• Non-idempotent POST without deduplication—risk double charge.
• Timeouts where server may have succeeded—prefer query status endpoint before blind retry.

@Retry(name = "paymentClient", fallbackMethod = "payFallback")
public PaymentResult charge(PaymentRequest req) {
    return paymentClient.charge(req);
}

// application.yml
// resilience4j.retry.instances.paymentClient.maxAttempts: 3
// waitDuration: 200ms, exponentialBackoffMultiplier: 2, enableRandomizedWait: true
// retryExceptions: IOException, HttpServerErrorException
// ignoreExceptions: HttpClientErrorException

Spring Retry (@Retryable, @Recover) remains in legacy codebases; new Spring Cloud projects standardize on Resilience4j for unified metrics and composition with circuit breakers.

States

• CLOSED — normal operation; failures are counted.
• OPEN — calls fail immediately (or invoke fallback); no load on dependency.
• HALF-OPEN — limited trial calls; success closes breaker, failure reopens.

stateDiagram-v2
  direction LR
  [*] --> Closed
  Closed --> Open: failure rate exceeds threshold
  Open --> HalfOpen: wait interval elapsed
  HalfOpen --> Closed: probe calls succeed
  HalfOpen --> Open: probe calls fail
  Closed --> Closed: calls succeed

Tuning metrics

@CircuitBreaker(name = "inventory", fallbackMethod = "defaultStock")
public StockLevel getStock(String sku) {
    return inventoryClient.fetch(sku);
}

private StockLevel defaultStock(String sku, Throwable t) {
    return StockLevel.unknown(sku);
}

Register event consumers on CircuitBreakerRegistry to log state transitions and export to Micrometer. Dashboards should show OPEN duration—long OPEN states mean dependency or config problems, not “breaker doing its job” forever.

Without bulkheads, all outbound calls share one thread pool (or one reactive event loop stall pattern). When Recommendations hangs, every thread waits on Recommendations and Catalog queries starve—even though Catalog is healthy.

Thread pool bulkhead

Dedicated executor per dependency or per subsystem. Calls submit work to the pool; when queue is full, fail fast. Resilience4j ThreadPoolBulkhead integrates with CompletableFuture-style APIs. Cost: thread overhead—size pools deliberately, not “max threads = 200” everywhere.

Semaphore bulkhead

Limits concurrent in-flight calls without extra threads—good for reactive stacks. Resilience4j Bulkhead with maxConcurrentCalls and maxWaitDuration (zero wait = immediate reject when full).

resilience4j.bulkhead:
  instances:
    recommendations:
      maxConcurrentCalls: 25
      maxWaitDuration: 0

resilience4j.thread-pool-bulkhead:
  instances:
    legacySoap:
      maxThreadPoolSize: 10
      coreThreadPoolSize: 4
      queueCapacity: 20

Algorithms compared

Resilience4j RateLimiter — per-instance in JVM; good for protecting one service’s calls to a fragile dependency. Spring Cloud Gateway RequestRateLimiter + Redis — distributed limit by user ID or API key at the edge. Redis Lua scripts atomically decrement counters for cluster-wide consistency.

spring:
  cloud:
    gateway:
      routes:
        - id: orders
          uri: lb://order-service
          filters:
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 50
                redis-rate-limiter.burstCapacity: 100
                key-resolver: "#{@userKeyResolver}"

Common patterns

• Return cached data — stale recommendations beat blank homepage; stamp with cachedAt for transparency internally.
• Return default / empty — empty related products list; zero balance placeholder with banner in UI.
• Fail fast — propagate 503 with clear message when operation cannot proceed (payment must not silently skip).
• Graceful degradation — core checkout works; non-critical features disabled via feature flags.

When fallback is not enough — compensation

Fallbacks cover read paths well. Write paths need sagas and compensating transactions: if Payment succeeded but Order commit failed, issue refund or mark order PENDING_REVIEW for manual reconciliation. Never hide failed writes behind silent fallbacks—financial and inventory domains require explicit state machines.

Add resilience4j-spring-boot3 and spring-boot-starter-aop. Annotations (@CircuitBreaker, @Retry, @Bulkhead, @RateLimiter, @TimeLimiter) apply via AOP around service methods or Feign clients.

Recommended execution order (outer → inner)

1. RateLimiter — reject overload before spending threads.
2. CircuitBreaker — fail fast if dependency unhealthy.
3. Bulkhead — cap concurrent calls.
4. TimeLimiter — bound wait time.
5. Retry — retry only the innermost business call (controversial: some teams place retry outside breaker; document team standard).

flowchart TB
  IN[Incoming call] --> RL[RateLimiter]
  RL --> CB[CircuitBreaker]
  CB --> BH[Bulkhead]
  BH --> TL[TimeLimiter]
  TL --> RT[Retry]
  RT --> SVC[Remote service]

@RateLimiter(name = "catalog")
@CircuitBreaker(name = "catalog", fallbackMethod = "catalogFallback")
@Bulkhead(name = "catalog")
@Retry(name = "catalog")
public ProductPage loadProduct(String sku) {
    return catalogClient.getProduct(sku);
}

Actuator and metrics

Expose /actuator/circuitbreakers, /actuator/ratelimiters, and /actuator/metrics/resilience4j.circuitbreaker.calls (with Prometheus registry). Alert on sustained OPEN state and rising not_permitted_calls from bulkheads—signs config is too tight or dependency degraded.

Hystrix provided thread-pool isolation, fallbacks, and the famous dashboard. Spring Cloud Netflix integrated via @HystrixCommand. Limitations that drove migration: blocking thread model overhead, no first-class reactive integration, and Netflix stopped active development.

Migration path: map Hystrix command groups to Resilience4j named instances; replace dashboard with Grafana + Micrometer; move edge resilience to Envoy/Istio where appropriate. Do not start new Hystrix projects in 2026.

Load testing with slow/fault-injecting dependencies (Toxiproxy, Litmus, Istio fault injection) verifies threads release under timeout and breakers open under error rate. Chaos engineering (controlled pod kills, network partition) validates fallbacks and sagas—not just happy-path unit tests with mocks.

Metrics to watch

• Breaker state transitions per dependency
• Retry count and exhausted retries (signals flaky network or bad thresholds)
• Bulkhead rejected calls
• Rate limiter wait time and timeouts
• End-to-end latency p99 on user journeys vs sum of internal calls

Tie thresholds to SLOs from Observability: if Checkout p99 budget is 800 ms and Payment p99 is 400 ms, Inventory cannot use a 2 s timeout without eating the entire budget. Revisit after every major traffic pattern change (Black Friday, product launch).