Circuit Breakers and Bulkheads

In a system built from many interdependent services, a slow or failing downstream dependency can take down callers that depend on it, and those callers can in turn take down their own callers — a cascading failure that turns one component's outage into a system-wide one. Circuit breakers and bulkheads are two complementary resilience patterns, borrowed from electrical engineering and ship design respectively, that contain failure at its source instead of letting it propagate. Neither pattern fixes the underlying failing dependency; both exist to protect the rest of the system while that dependency is unhealthy.

A circuit breaker wraps calls to a remote dependency and tracks their success and failure rate. In the closed state, calls pass through normally while failures are counted. Once failures cross a configured threshold (e.g. more than 50% of calls fail within a rolling window), the breaker trips to the open state, in which calls fail immediately without even attempting the network call — this is the critical protective effect, since it stops wasting threads, connections, and time waiting on a dependency that is already known to be unhealthy, and it stops piling additional load onto a struggling dependency. After a cooldown period, the breaker moves to half-open, allowing a small number of trial calls through; if they succeed, the breaker closes again and normal traffic resumes, and if they fail, it reopens and the cooldown restarts.

A bulkhead partitions a shared resource — typically a thread pool or connection pool — into isolated segments, one per downstream dependency, so that if calls to one dependency start hanging, they exhaust only their own segment's threads and leave capacity for calls to every other dependency untouched. Without bulkheading, a single slow dependency sharing a common thread pool with everything else can consume every available thread waiting on it, starving unrelated, perfectly healthy requests that have nothing to do with the failing dependency — this is sometimes called resource exhaustion cascading through an otherwise unrelated code path.

Circuit breakers and bulkheads address different failure modes and are typically used together: bulkheads prevent one dependency's resource exhaustion from starving calls to other dependencies, while circuit breakers stop wasting even a bulkhead's own limited resources on calls to a dependency that is already known to be failing. In combination, a call to a downstream service first checks whether that dependency's circuit breaker is open (fail fast if so), and if closed, executes within that dependency's dedicated bulkhead, so no single dependency's failure — cascading resource exhaustion or otherwise — can degrade the rest of the system.