UNDER PRESSURE

Level 3 · Systems That Depend on Each Other

Session 25: The Domino Effect: Cascading Failure, Timeout, & Circuit Breaker

What if one small microservice slowing down for 10 seconds managed to drain the entire connection pool and every worker thread across 20 upstream servers, bringing down the whole e-commerce platform in under 2 minutes?

Session 25 / 344 min read

1. Imagine if

On a giant cargo ship weighing hundreds of thousands of tons, the hull isn't designed as one huge empty space. Instead, it's divided into a dozen or more watertight compartments (Bulkheads). If the hull hits a reef and one compartment springs a leak and fills with water, the bulkhead doors close tight automatically. Only that compartment gets wet, while the ship keeps floating normally.

Now imagine the ship had no bulkheads at all: one tiny nail hole at the far end of the stern would let water flow slowly into every room, kill the main engine, and sink the entire ship. That's Cascading Failure in a distributed system.

2. What actually happens

When a downstream service (for example: Payment Service or a Third-Party SMS Provider) runs into trouble:

  1. Thread Starvation & Pool Exhaustion: - The upstream service (Order Service) sends a request to Payment. - Because there's no time limit (No Timeout) or the timeout is set too loosely (say 30 seconds), worker threads in Order Service are stuck waiting (blocking IO). - When 200 requests come in at once, all 200 threads in Order Service end up tied up waiting for Payment to reply. - The result: Order Service stops responding to other requests (even read requests that don't need Payment at all).

  2. The Domino Effect Spreads: - Because Order Service is frozen, the API Gateway calling Order Service runs out of connections too. - Because the API Gateway is frozen, the load balancer decides the whole cluster is down, and users see a blank white 504 Gateway Timeout error screen. - One small, slow service sinks the entire platform.

3. The Lifesavers: Circuit Breaker & Bulkhead

To break this chain of destruction, architects install a Circuit Breaker (like the electrical fuse in your house):

  1. The Three Circuit Breaker States: - CLOSED (Normal): All requests flow through as usual. The system tracks the error/timeout ratio. - OPEN (Tripped / Cut): If failures cross the threshold (say 50% of requests failing/timing out within 10 seconds), the circuit BLOWS immediately. Every following request is rejected instantly (Fail-Fast in 0 ms) without ever touching the dying downstream! - HALF-OPEN (Trial): After a cool-down period (say 30 seconds), the circuit lets a few trial requests (probes) through. If they succeed, the circuit goes back to CLOSED. If they fail, it goes back to OPEN.

  2. Bulkhead Pattern (Thread Pool Isolation): - Split thread pool allocation per downstream service (for example: 50 threads for Catalog, 10 threads for the SMS Provider). - If the SMS Provider jams completely, at most 10 threads get locked up; the 50 Catalog threads keep working 100% normally.

4. The official name

  • Cascading Failure: A chain of failures where the death of one component triggers overload and the death of other components.
  • Circuit Breaker Pattern: A resilience design that stops executing operations that keep failing, in order to protect the system.
  • Fail-Fast: An architectural principle where the system returns a failure response as quickly as possible instead of leaving the client waiting.
  • Bulkhead Pattern: A pattern for isolating resources (threads, memory, connection pools) so that a failure in one area doesn't spread to others.

5. In our world

  • Netflix Hystrix / Resilience4j: Industry-standard libraries for wrapping RPC calls with a Circuit Breaker and Bulkhead.
  • Envoy Proxy / Istio Service Mesh: Provide automatic Circuit Breaking at the infrastructure layer without changing a single line of application code.

6. The performance tester's lens

Crucial metrics and tests: 1. Trip Threshold Verification: Inject 5 seconds of latency into a mock payment service under a 500 QPS load. Verify that the Circuit Breaker switches to the OPEN state within <3 seconds. 2. Fail-Fast Latency Check: While in the OPEN state, make sure the fallback error response comes back in <2 milliseconds, not tens of seconds. 3. Half-Open Recovery: Restore the mock payment service and make sure the system automatically returns to the CLOSED state without needing an application restart.

7. Question for the next round

When a downstream service returns an error, a developer's first instinct is often: "Try again! (Retry!)".

But what happens if 10,000 users' smartphones try to retry a failed request all at the same time, right as the server has just come back up?

That disaster is called a Retry Storm. The answer is in Session 26: When Everyone Tries Again: Retry Storms, Jitter, & Backoff.