Designing Fault Isolation with Bulkheads

Resilience is not a feature you add at the end — it is mauri, the life force of your infrastructure. Bulkhead patterns isolate resources so a failure in one area cannot exhaust capacity across the entire system.

Who Benefits

Platform and SRE teams gain predictable blast radius during incidents. Development teams can deploy independently without fear that one service’s memory leak will take down neighbours.

The Problem

In a shared pool architecture, a spike in traffic or a slow dependency can consume all available threads, connections, or memory. One misbehaving component degrades everything connected to it.

The Bulkhead Pattern

Named after ship compartments that contain flooding, bulkheads partition resources:

• Thread pools — Separate pools per downstream dependency
• Connection limits — Cap concurrent connections to any single service
• Process isolation — Run critical and non-critical workloads on separate compute
• Deployment boundaries — Isolate failure domains by region or availability zone

Decision Criteria

Implementation Steps

1. Identify failure domains — Map which components share threads, pools, or nodes.
2. Set resource quotas — Define hard limits per domain (CPU, memory, connections).
3. Add backpressure — Reject or queue requests when a bulkhead fills rather than blocking indefinitely.
4. Monitor per-bulkhead metrics — Track utilisation, rejection rate, and queue depth separately.
5. Test isolation — Chaos experiments that overload one bulkhead should not affect others.

Verification

• Load-test one service to exhaustion; neighbouring services maintain SLOs.
• Alerts fire on bulkhead saturation before user-visible degradation.
• Runbooks document which bulkhead failed and how to restore capacity.

Related Patterns

• Circuit breaker — Stop calling a failing dependency entirely
• Retry with jitter — Avoid retry storms that refill a saturated bulkhead
• Rate limiting — Protect upstream services from overload