Circuit Breakers and Bulkheads
In a system built from many interdependent services, a slow or failing downstream dependency can take down callers that depend on it, and those callers can in turn take down their own callers — a cascading failure that turns one component's outage into a system-wide one. Circuit breakers and bulkheads are two complementary resilience patterns, borrowed from electrical engineering and ship design respectively, that contain failure at its source instead of letting it propagate. Neither pattern fixes the underlying failing dependency; both exist to protect the rest of the system while that dependency is unhealthy.
Cricket analogy: One batter's slow, panicked innings can pressure the next batter into rash shots trying to catch up on the required rate, and that panic can spread down the order — a single failing partnership cascading into a full collapse.
Circuit Breaker State Machine
A circuit breaker wraps calls to a remote dependency and tracks their success and failure rate. In the closed state, calls pass through normally while failures are counted. Once failures cross a configured threshold (e.g. more than 50% of calls fail within a rolling window), the breaker trips to the open state, in which calls fail immediately without even attempting the network call — this is the critical protective effect, since it stops wasting threads, connections, and time waiting on a dependency that is already known to be unhealthy, and it stops piling additional load onto a struggling dependency. After a cooldown period, the breaker moves to half-open, allowing a small number of trial calls through; if they succeed, the breaker closes again and normal traffic resumes, and if they fail, it reopens and the cooldown restarts.
Cricket analogy: A captain tracks a bowler's economy rate over (closed state, bowling continues normally); once it crosses a bad threshold across an over, the captain pulls them from the attack entirely (open state, no more overs bowled) rather than keep leaking runs, later giving them one trial over to see if they've settled (half-open) before fully reinstating them.
Circuit breaker state machine
failure rate > threshold
CLOSED ------------------------> OPEN
^ |
| trial calls succeed | cooldown timer elapses
| v
+----------------------- HALF-OPEN
trial calls fail: back to OPEN
CLOSED: calls pass through; failures counted
OPEN: calls fail fast, no network attempt made
HALF-OPEN: a few trial calls allowed to test recoveryBulkheads: Isolating Resource Pools
A bulkhead partitions a shared resource — typically a thread pool or connection pool — into isolated segments, one per downstream dependency, so that if calls to one dependency start hanging, they exhaust only their own segment's threads and leave capacity for calls to every other dependency untouched. Without bulkheading, a single slow dependency sharing a common thread pool with everything else can consume every available thread waiting on it, starving unrelated, perfectly healthy requests that have nothing to do with the failing dependency — this is sometimes called resource exhaustion cascading through an otherwise unrelated code path.
Cricket analogy: A cricket board splits its total broadcast-production budget into separate pools per tournament, so if one tournament's production runs wildly over budget, it can't drain the funds reserved for an unrelated tournament happening the same month.
Netflix's Hystrix library, which pioneered widely-adopted circuit breaker and bulkhead patterns for microservices, famously used per-dependency thread pools sized deliberately small (e.g. 10 threads) specifically so that a hanging dependency could only ever exhaust its own small pool, never the shared application thread pool serving all other traffic. Although Hystrix itself is now in maintenance mode, its patterns live on in libraries like resilience4j.
A common mistake is setting a circuit breaker's failure threshold and cooldown period identically for every dependency regardless of its criticality and typical latency profile. A fast, low-latency internal cache lookup and a slow third-party payment gateway have very different normal failure/latency baselines, and using one-size-fits-all thresholds either trips the breaker on the payment gateway's normal behavior or fails to protect against a genuinely broken cache.
Combining the Two Patterns
Circuit breakers and bulkheads address different failure modes and are typically used together: bulkheads prevent one dependency's resource exhaustion from starving calls to other dependencies, while circuit breakers stop wasting even a bulkhead's own limited resources on calls to a dependency that is already known to be failing. In combination, a call to a downstream service first checks whether that dependency's circuit breaker is open (fail fast if so), and if closed, executes within that dependency's dedicated bulkhead, so no single dependency's failure — cascading resource exhaustion or otherwise — can degrade the rest of the system.
Cricket analogy: Before sending a bowler out for another over, a captain first checks whether that bowler has been pulled from the attack entirely (the circuit breaker check), and if not, that bowler still only has their own allotted overs to work with (the bulkhead) — so no single bowler's bad day can consume the whole team's over allocation.
- Circuit breakers wrap remote calls and fail fast once a failure-rate threshold is crossed, avoiding wasted time and load on a known-unhealthy dependency.
- The breaker's three states are closed (normal), open (fail fast), and half-open (trial calls to test recovery).
- Bulkheads partition shared resources like thread pools per dependency so one hanging dependency cannot exhaust capacity needed by others.
- Netflix's Hystrix popularized both patterns together, using deliberately small per-dependency thread pools as bulkheads.
- Threshold and cooldown settings should be tuned per dependency based on its normal latency and failure baseline, not applied uniformly.
- Circuit breakers and bulkheads are complementary: bulkheads isolate resource exhaustion, while breakers stop calls before they even consume bulkhead resources.
Practice what you learned
1. What happens when a circuit breaker is in the open state?
2. What is the purpose of the half-open state in a circuit breaker?
3. What problem does a bulkhead solve that a circuit breaker alone does not?
4. Why did Netflix's Hystrix use deliberately small, per-dependency thread pools?
5. Why is it a mistake to apply identical failure thresholds and cooldowns to every dependency's circuit breaker?
Was this page helpful?
You May Also Like
Rate Limiting Strategies
Techniques for controlling how many requests a client or service can make in a given time window, protecting systems from overload and abuse.
Redundancy and Failover
Techniques for eliminating single points of failure by duplicating critical components and automatically routing traffic away from failed instances to healthy ones.
Monitoring and Observability
The practices and tooling — metrics, logs, and traces — that let engineers understand a distributed system's internal state from its external outputs and detect or diagnose problems.
Idempotency in Distributed Systems
The property that performing the same operation multiple times produces the same result as performing it once, essential for safely retrying requests over unreliable networks.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesMicroservices Study Notes
Software Architecture · 30 topics
Software EngineeringTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 75 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics