Designing Fault-Tolerant Systems: Circuit Breakers, Retries & Bulkheads

In the complex landscape of modern distributed systems, especially microservices architectures, failure is not an exception but an inevitability. Network glitches, service overloads, database timeouts, and unexpected bugs can all conspire to bring down a single service. Without proper safeguards, a minor hiccup in one component can quickly cascade into a catastrophic outage across an entire system. This is where the principles of fault tolerance become paramount.

As a senior engineer, I’ve seen firsthand how a well-architected system, even under immense stress, can remain resilient, serving its users effectively. Conversely, I’ve also witnessed the domino effect of a poorly designed system where a single point of failure brought everything to a grinding halt. This article will dive deep into three fundamental patterns for building fault-tolerant systems: Retries, Circuit Breakers, and Bulkheads. We’ll explore their individual mechanisms, how they complement each other, provide practical code examples, and discuss real-world architectural implications.

The Inevitability of Failure in Distributed Systems

Think about a typical microservices ecosystem. A single user request might traverse multiple services: an API Gateway, an authentication service, a product catalog service, a recommendation engine, an inventory service, and finally, a payment gateway. Each of these services relies on networks, databases, caches, and potentially other external APIs. The probability of one of these components experiencing a momentary issue increases exponentially with the number of dependencies.

Without fault tolerance, a transient network error between Service A and Service B could cause Service A to hang, consuming resources. If many requests hit Service A simultaneously, all waiting for Service B, Service A’s resources (threads, connections, memory) could be exhausted, leading to its own failure. This failure then propagates upstream to the API Gateway, and eventually, the user sees an error. This chain reaction is known as a cascading failure, and it’s the nightmare of every operations team.

Our goal is to design systems that are resilient to these failures, allowing them to degrade gracefully rather than collapsing entirely. This means anticipating problems, isolating them, and providing mechanisms for recovery or alternative paths. Let’s explore the tools that help us achieve this.

Fault-Tolerant Systems: Circuit Breakers, Retries & Bulkheads — Generated Image

1. Retries: Giving Operations a Second Chance

The simplest and often first line of defense against transient failures is the retry mechanism. Many errors in distributed systems are temporary: a momentary network blip, a database deadlock, a service briefly becoming unavailable during a redeployment, or a rate limit being momentarily hit. In such cases, simply retrying the failed operation after a short delay can lead to success.

When to Use Retries

Transient Errors: Ideal for operations that are likely to succeed on a subsequent attempt, such as network timeouts, connection resets, or temporary service unavailability (e.g., HTTP 500, 503 errors).
Idempotent Operations: Crucially, retries should primarily be applied to idempotent operations. An operation is idempotent if executing it multiple times has the same effect as executing it once. For example, updating a user’s address is often idempotent (setting the address to ‘X’ multiple times results in the address being ‘X’). However, deducting money from an account is generally not (doing it twice deducts twice the amount). For non-idempotent operations, retries can lead to unintended side effects.

Retry Strategies

Blindly retrying immediately can often exacerbate the problem, especially if the downstream service is overloaded. Therefore, various strategies are employed to make retries more effective and less harmful:

Fixed Delay Retry: The simplest approach, where an operation is retried after a constant delay (e.g., 2 seconds) for a fixed number of attempts. While easy to implement, it might not be optimal if the downstream service needs more time to recover.

// Fixed Delay Retry (Conceptual)
void callServiceFixedDelay(Supplier<Result> operation, int maxAttempts, long delayMillis) {
    for (int i = 0; i < maxAttempts; i++) {
        try {
            operation.get(); // Execute the operation
            return; // Success
        } catch (Exception e) {
            if (i == maxAttempts - 1) throw e; // Last attempt, rethrow
            Thread.sleep(delayMillis); // Wait before retrying
        }
    }
}

Exponential Backoff Retry: This is a much preferred strategy. The delay between retries increases exponentially with each subsequent attempt (e.g., 1s, 2s, 4s, 8s). This gives the downstream service more time to recover from a potentially overloaded state. It prevents the retrying service from hammering the struggling service repeatedly.

// Exponential Backoff Retry (Conceptual)
void callServiceExponentialBackoff(Supplier<Result> operation, int maxAttempts, long initialDelayMillis) {
    long currentDelay = initialDelayMillis;
    for (int i = 0; i < maxAttempts; i++) {
        try {
            operation.get(); // Execute the operation
            return; // Success
        } catch (Exception e) {
            if (i == maxAttempts - 1) throw e; // Last attempt, rethrow
            Thread.sleep(currentDelay); // Wait
            currentDelay *= 2; // Double the delay for next attempt
        }
    }
}

Exponential Backoff with Jitter: While exponential backoff is good, if many clients retry simultaneously using the exact same backoff schedule, they might all retry at the same time, creating a “thundering herd” problem. Jitter adds a small, random deviation to the calculated delay. For example, instead of waiting exactly 2 seconds, it might wait between 1.8 and 2.2 seconds. This disperses the retries, reducing the chances of simultaneous requests hitting the recovering service.

// Exponential Backoff with Jitter (Conceptual)
import java.util.Random;

void callServiceExponentialBackoffWithJitter(Supplier<Result> operation, int maxAttempts, long initialDelayMillis) {
    long currentDelay = initialDelayMillis;
    Random random = new Random();

    for (int i = 0; i < maxAttempts; i++) {
        try {
            operation.get(); // Execute the operation
            return; // Success
        } catch (Exception e) {
            if (i == maxAttempts - 1) throw e; // Last attempt, rethrow

            long jitter = (long) (random.nextDouble() * currentDelay / 2); // Add/subtract up to 50% of currentDelay
            long sleepTime = currentDelay / 2 + jitter; // A common strategy: delay = base + random(0, base) or base/2 + random(0, base/2)

            Thread.sleep(sleepTime); // Wait
            currentDelay *= 2; // Double the base delay for next attempt
        }
    }
}

Considerations for Retries

Retry Limits: Always set a maximum number of retries to prevent an operation from blocking indefinitely.
Timeouts: Combine retries with timeouts. Each individual attempt should have a reasonable timeout. If the downstream service is consistently slow, even retries won’t help and might just prolong the suffering.
Circuit Breakers Integration: Retries should ideally be used in conjunction with circuit breakers. If a service is consistently failing, retries will just waste resources until the circuit breaker trips.
Client-side vs. Server-side: Retries can be implemented at the client-side (caller) or by an intermediate proxy/load balancer.

While retries are powerful for transient issues, they are not a silver bullet. If a service is down or completely overwhelmed, endless retries will only make things worse. This is where the circuit breaker pattern comes into play.

2. Circuit Breakers: Preventing Cascading Failures

Imagine an electrical circuit breaker in your home. When there’s an overload or a short circuit, it “trips,” opening the circuit to prevent damage to appliances or the wiring. The circuit remains open until the fault is resolved and it’s manually reset. The software circuit breaker pattern works similarly.

A circuit breaker wraps a protected function call (e.g., a call to an external service). It monitors for failures. If the failure rate crosses a certain threshold within a specified time window, the circuit “trips” or “opens,” preventing further calls to the failing service. Instead of attempting the call and waiting for it to fail, the circuit breaker immediately returns an error or a fallback response, thus failing fast. After a configurable timeout, it allows a limited number of “test” calls to see if the service has recovered, entering a “half-open” state. If these test calls succeed, the circuit “closes,” allowing normal traffic again. If they fail, it re-opens.

States of a Circuit Breaker

Closed: This is the default state. Calls to the protected service are allowed to pass through. The circuit breaker monitors for failures. If the failure rate exceeds a configured threshold, it transitions to the Open state.
Open: In this state, the circuit breaker immediately rejects all calls to the protected service without attempting them. It returns an error (e.g., a specific exception) or a predefined fallback response. This protects the failing service from further load and prevents the calling service from wasting resources waiting for timeouts. After a configured waitDurationInOpenState (often called “reset timeout”), it transitions to the Half-Open state.
Half-Open: In this state, the circuit breaker allows a limited number of test calls to pass through to the protected service. If these test calls succeed, it’s an indication that the service might have recovered, and the circuit transitions back to Closed. If these test calls fail, the service is still unhealthy, and the circuit immediately transitions back to Open, restarting the waitDurationInOpenState.

Benefits of Circuit Breakers

Fail Fast: Instead of waiting for a timeout, the calling service immediately knows the downstream service is unhealthy, preventing resource exhaustion.
Prevents Cascading Failures: By stopping traffic to a failing service, it gives that service time to recover and prevents its unresponsiveness from spreading upstream.
Graceful Degradation: Allows for implementing fallback logic, providing a degraded but still functional experience to the user instead of a complete failure.
Reduced Load on Failing Service: Protects the struggling service from being overwhelmed by continuous retry attempts, giving it space to heal.

Implementing Circuit Breakers (with Resilience4j)

In the Java ecosystem, Resilience4j is a popular library for implementing fault tolerance patterns, including Circuit Breakers. Let’s look at a simplified example.

import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;
import io.vavr.CheckedFunction0;
import io.vavr.control.Try;

import java.time.Duration;

public class CircuitBreakerExample {

    private final CircuitBreaker circuitBreaker;

    public CircuitBreakerExample() {
        // 1. Configure the Circuit Breaker
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
                .failureRateThreshold(50) // If 50% of calls fail...
                .waitDurationInOpenState(Duration.ofSeconds(5)) // ...stay open for 5 seconds
                .permittedNumberOfCallsInHalfOpenState(3) // Then allow 3 test calls
                .slidingWindowSize(10) // Evaluate last 10 calls
                .build();

        // 2. Create a Circuit Breaker Registry
        CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);

        // 3. Get or create a Circuit Breaker instance
        circuitBreaker = registry.circuitBreaker("myServiceCircuitBreaker");

        // Optional: Attach event listeners for monitoring
        circuitBreaker.getEventPublisher()
                .onStateTransition(event -> System.out.println("CircuitBreaker State Transition: " + event.getOldState() + " -> " + event.getNewState()));
        circuitBreaker.getEventPublisher()
                .onError(event -> System.out.println("CircuitBreaker Error: " + event.getThrowable().getMessage()));
    }

    // Simulate a service call that sometimes fails
    private String callExternalService() {
        // Simulate a 70% failure rate
        if (Math.random() < 0.7) {
            System.out.println("  -> External Service call FAILED!");
            throw new RuntimeException("External Service Unavailable!");
        }
        System.out.println("  -> External Service call SUCCEEDED!");
        return "Data from External Service";
    }

    public String makeProtectedCall() {
        // 4. Decorate the service call with the Circuit Breaker
        CheckedFunction0<String> decoratedCall = CircuitBreaker.decorateCheckedSupplier(circuitBreaker, this::callExternalService);

        // 5. Execute the decorated call with a fallback
        return Try.of(decoratedCall)
                .recover(throwable -> {
                    System.out.println("  -> Fallback executed due to: " + throwable.getMessage());
                    return "Fallback Data";
                })
                .get();
    }

    public static void main(String[] args) throws InterruptedException {
        CircuitBreakerExample example = new CircuitBreakerExample();

        System.out.println("--- Initial calls (expect some failures, then open) ---");
        for (int i = 0; i < 20; i++) {
            System.out.println("Attempt " + (i + 1) + ": " + example.makeProtectedCall());
            Thread.sleep(100); // Small delay between calls
        }

        System.out.println("\n--- Circuit is OPEN (expect mostly fallbacks) ---");
        for (int i = 0; i < 10; i++) {
            System.out.println("Attempt " + (i + 21) + ": " + example.makeProtectedCall());
            Thread.sleep(100);
        }

        System.out.println("\n--- Waiting for 5 seconds (Open state timeout) ---");
        Thread.sleep(Duration.ofSeconds(5).toMillis()); // Wait for the circuit breaker to transition to Half-Open

        System.out.println("\n--- Circuit is HALF-OPEN (expect a few test calls, then close or re-open) ---");
        for (int i = 0; i < 10; i++) {
            System.out.println("Attempt " + (i + 31) + ": " + example.makeProtectedCall());
            Thread.sleep(100);
        }

        System.out.println("\n--- After Half-Open attempts ---");
        for (int i = 0; i < 5; i++) {
            System.out.println("Attempt " + (i + 41) + ": " + example.makeProtectedCall());
            Thread.sleep(100);
        }
    }
}

The example demonstrates how to configure a circuit breaker with failure thresholds, wait duration in the open state, and the number of permitted calls in the half-open state. It then wraps a simulated external service call. When the service fails frequently, the circuit opens, and subsequent calls immediately return fallback data. After the wait duration, it enters half-open, making a few test calls to determine if the service has recovered.

Circuit breakers are crucial for system stability, but they don’t solve the problem of resource exhaustion if a service is merely slow, not completely down. For that, we turn to bulkheads.

3. Bulkheads: Isolating Resource Failures

The bulkhead pattern comes from shipbuilding. Ships are divided into watertight compartments (bulkheads) so that if one compartment is breached and floods, the entire ship doesn’t sink. In software, the bulkhead pattern isolates parts of the system so that a failure or degradation in one part does not impact the entire system.

The primary goal of bulkheads is to prevent resource exhaustion. If a particular downstream service becomes slow, requests to that service might start consuming all available threads, memory, or network connections in the calling service, leading to its own collapse. Bulkheads address this by allocating a fixed, limited pool of resources for calls to specific external services or components.

Types of Bulkheads

Thread Pool Bulkheads: This is the most common implementation. Each downstream service or critical operation is assigned its own dedicated thread pool. If calls to Service X become slow, only the thread pool allocated for Service X will be exhausted, leaving other thread pools (and thus other operations/services) unaffected.

This is particularly useful when you have services with varying response times or reliability. A slow external payment gateway shouldn’t bring down your internal product catalog service.

import io.github.resilience4j.bulkhead.ThreadPoolBulkhead;
import io.github.resilience4j.bulkhead.ThreadPoolBulkheadConfig;
import io.github.resilience4j.bulkhead.ThreadPoolBulkheadRegistry;
import io.vavr.control.Try;

import java.time.Duration;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;

public class ThreadPoolBulkheadExample {

    private final ThreadPoolBulkhead serviceABulkhead;
    private final ThreadPoolBulkhead serviceBBulkhead;

    public ThreadPoolBulkheadExample() {
        // Configure Bulkhead for Service A
        ThreadPoolBulkheadConfig configA = ThreadPoolBulkheadConfig.custom()
                .maxThreadPoolSize(5) // Max 5 threads for Service A calls
                .coreThreadPoolSize(2) // Keep 2 threads always alive
                .queueCapacity(10) // Queue up to 10 requests if threads are busy
                .keepAliveTime(Duration.ofSeconds(1))
                .build();

        // Configure Bulkhead for Service B
        ThreadPoolBulkheadConfig configB = ThreadPoolBulkheadConfig.custom()
                .maxThreadPoolSize(3) // Max 3 threads for Service B calls
                .coreThreadPoolSize(1)
                .queueCapacity(5) // Queue up to 5 requests
                .keepAliveTime(Duration.ofSeconds(1))
                .build();

        ThreadPoolBulkheadRegistry registry = ThreadPoolBulkheadRegistry.ofDefaults();
        serviceABulkhead = registry.bulkhead("serviceA", configA);
        serviceBBulkhead = registry.bulkhead("serviceB", configB);

        // Optional: Attach event listeners
        serviceABulkhead.getEventPublisher()
                .onCallRejected(event -> System.out.println("Service A Bulkhead REJECTED a call!"));
        serviceBBulkhead.getEventPublisher()
                .onCallRejected(event -> System.out.println("Service B Bulkhead REJECTED a call!"));
    }

    private String callServiceA() throws InterruptedException {
        Thread.sleep(500); // Simulate network latency/processing time for Service A
        return "Response from Service A";
    }

    private String callServiceB() throws InterruptedException {
        Thread.sleep(1500); // Simulate a much slower Service B
        return "Response from Service B";
    }

    public Future<String> makeServiceACall() {
        Supplier<Future<String>> decoratedSupplier = serviceABulkhead.decorateSupplier(
                () -> {
                    try {
                        return Executors.newSingleThreadExecutor().submit(this::callServiceA);
                    } catch (Exception e) {
                        throw new RuntimeException(e);
                    }
                });

        return Try.of(decoratedSupplier)
                .recover(throwable -> {
                    System.out.println("  -> Service A call failed/rejected: " + throwable.getMessage());
                    return Executors.newSingleThreadExecutor().submit(() -> "Fallback for Service A");
                })
                .get();
    }

    public Future<String> makeServiceBCall() {
        Supplier<Future<String>> decoratedSupplier = serviceBBulkhead.decorateSupplier(
                () -> {
                    try {
                        return Executors.newSingleThreadExecutor().submit(this::callServiceB);
                    } catch (Exception e) {
                        throw new RuntimeException(e);
                    }
                });

        return Try.of(decoratedSupplier)
                .recover(throwable -> {
                    System.out.println("  -> Service B call failed/rejected: " + throwable.getMessage());
                    return Executors.newSingleThreadExecutor().submit(() -> "Fallback for Service B");
                })
                .get();
    }

    public static void main(String[] args) throws InterruptedException {
        ThreadPoolBulkheadExample example = new ThreadPoolBulkheadExample();

        System.out.println("--- Making multiple concurrent calls ---");
        // Simulate many calls to Service A and Service B concurrently
        for (int i = 0; i < 15; i++) {
            final int requestNum = i;
            // Service A calls (faster, more capacity)
            new Thread(() -> {
                Future<String> resultFutureA = example.makeServiceACall();
                try {
                    System.out.println("Request " + requestNum + " (A): " + resultFutureA.get(2, TimeUnit.SECONDS));
                } catch (Exception e) {
                    System.out.println("Request " + requestNum + " (A) timed out or failed: " + e.getMessage());
                }
            }).start();

            // Service B calls (slower, less capacity)
            new Thread(() -> {
                Future<String> resultFutureB = example.makeServiceBCall();
                try {
                    System.out.println("Request " + requestNum + " (B): " + resultFutureB.get(2, TimeUnit.SECONDS));
                } catch (Exception e) {
                    System.out.println("Request " + requestNum + " (B) timed out or failed: " + e.getMessage());
                }
            }).start();

            Thread.sleep(50); // Stagger requests
        }

        Thread.sleep(5000); // Give time for operations to complete
        System.out.println("\n--- Done ---");
    }
}

In this example, Service A has a larger thread pool and queue, while Service B has smaller resources. If Service B becomes extremely slow, its bulkhead will reject calls once its thread pool and queue are full, but Service A’s calls will continue to be processed without being affected.

Semaphore Bulkheads (Rate Limiting/Concurrency Limiting): Instead of managing separate thread pools, semaphore bulkheads limit the number of concurrent executions of an operation using a semaphore. If the semaphore count is reached, new requests are either rejected or queued. This is simpler to implement than thread pool bulkheads and often used for protecting specific critical sections or external calls where thread isolation isn’t strictly necessary but concurrency control is.

import io.github.resilience4j.bulkhead.Bulkhead;
import io.github.resilience4j.bulkhead.BulkheadConfig;
import io.github.resilience4j.bulkhead.BulkheadRegistry;
import io.vavr.control.Try;

import java.time.Duration;
import java.util.function.Supplier;

public class SemaphoreBulkheadExample {

    private final Bulkhead databaseBulkhead;

    public SemaphoreBulkheadExample() {
        // Configure Bulkhead for database calls
        BulkheadConfig config = BulkheadConfig.custom()
                .maxConcurrentCalls(5) // Allow max 5 concurrent calls
                .maxWaitDuration(Duration.ofMillis(100)) // Wait up to 100ms for a permit
                .build();

        BulkheadRegistry registry = BulkheadRegistry.of(config);
        databaseBulkhead = registry.bulkhead("databaseCalls");

        databaseBulkhead.getEventPublisher()
                .onCallRejected(event -> System.out.println("Database Bulkhead REJECTED a call!"));
    }

    private String callDatabase() throws InterruptedException {
        // Simulate a database call that takes some time
        long processingTime = 300 + (long) (Math.random() * 200); // 300-500ms
        System.out.println(Thread.currentThread().getName() + " -> Calling DB for " + processingTime + "ms");
        Thread.sleep(processingTime);
        return "DB Record";
    }

    public String makeProtectedDbCall() {
        Supplier<String> decoratedCall = Bulkhead.decorateSupplier(databaseBulkhead, this::callDatabase);

        return Try.of(decoratedCall)
                .recover(throwable -> {
                    System.out.println("  -> DB call failed/rejected: " + throwable.getMessage());
                    return "Fallback: Cached DB Record";
                })
                .get();
    }

    public static void main(String[] args) throws InterruptedException {
        SemaphoreBulkheadExample example = new SemaphoreBulkheadExample();

        System.out.println("--- Making multiple concurrent DB calls ---");
        for (int i = 0; i < 20; i++) {
            final int requestNum = i;
            new Thread(() -> {
                System.out.println("Request " + requestNum + ": " + example.makeProtectedDbCall());
            }, "Request-Thread-" + requestNum).start();
            Thread.sleep(50); // Stagger requests
        }

        Thread.sleep(3000); // Give time for operations to complete
        System.out.println("\n--- Done ---");
    }
}

Here, only 5 concurrent calls to the database are allowed. If more requests come in, they either wait briefly or are immediately rejected, protecting the database from being overwhelmed and ensuring other operations in the application are not starved of threads.

Considerations for Bulkheads

Granularity: Decide on the granularity of your bulkheads. Should it be per downstream service, per API endpoint, or per resource type (e.g., all database calls)?
Resource Allocation: Carefully size your thread pools or semaphore limits. Too small, and you’ll reject legitimate traffic; too large, and you risk resource exhaustion.
Overhead: Thread pool bulkheads have more overhead than semaphore bulkheads due to thread creation and management. Choose based on your specific needs.
Monitoring: Monitor bulkhead metrics (active calls, queued calls, rejected calls) to fine-tune configurations.

Putting it All Together: An Architectural Approach

These three patterns are not mutually exclusive; in fact, they are most effective when used in combination. Let’s consider a typical microservices architecture to see how they integrate.

Scenario: E-commerce Checkout Service

Imagine a user initiates a checkout process. The Checkout Service needs to:

Validate user session with Auth Service.
Retrieve product details from Product Catalog Service.
Check inventory with Inventory Service.
Process payment with Payment Gateway (external API).
Update order status in Order Service.
Notify Shipping Service.

Applying the Patterns:

1. Retries:

Apply exponential backoff retries to calls to Auth Service, Product Catalog Service, Inventory Service, and Order Service for transient errors (e.g., network issues, temporary database locks).
Crucially, ensure operations are idempotent (e.g., updating an order status can be idempotent, but processing payment usually isn’t without careful design).
For Payment Gateway, retries might be more complex, requiring careful consideration of transaction IDs and idempotency keys to prevent double charges.

2. Circuit Breakers:

Wrap every external service call with a Circuit Breaker.
Auth Service Circuit Breaker: If Auth fails, the circuit opens. Fallback could be to allow guest checkout or deny access with a “try again later”

Tags: bulkheadscircuit breakerfault tolerancekhadervalimicroservicesretries

Written by

Khader Vali

Senior Software Engineer specializing in cloud architecture, real-time systems, and enterprise-scale applications.

Share this article

Fault-Tolerant Systems: Circuit Breakers, Retries & Bulkheads