liveBook · Manning

Retry storm

As explained in Grokking Software Architecture

A retry storm is a burst of repeated requests generated by concurrent retries that overload an already struggling dependency or service.

A retry storm happens when concurrent retries pile onto a struggling service, multiplying load instead of restoring service. A two-second delay at a Payment API can trigger the problem: an Order Service mistakes the delay for a lost request and retries. section 12.8.3

When retries make things worse

With 1,000 users and a naive loop that makes three unspaced retries, the struggling Payment API receives 3,000 extra requests. The database can melt under this amplified traffic. section 12.8.3

Dampen the failure with two shock absorbers:

  • Use a circuit breaker to stop retrying and fail fast when a downstream service is struggling. section 12.8.3
  • Use exponential backoff with jitter to space retries and give the service time to recover. section 12.8.3
Figure 12.8 The Retry Storm. Without exponential backoff or a circuit breaker, a minor 2-second network delay causes naive loops to multiply traffic, accidentally launching a Distributed Denial of Service (DDoS) attack against your own database.

Shows how a minor delay and naive retry loops can multiply traffic into a self-inflicted denial-of-service attack. section 12.8.3

How Grokking Software Architecture covers it

How deep it goes
Defines the failure through a payment-call scenario and prescribes mitigations without detailing their implementation.
Examples
Uses a delayed Payment API, 1,000 concurrent users, and 3,000 unspaced requests to illustrate traffic amplification.
What you can do after
Diagnose retry amplification and mitigate it with circuit breakers or exponential backoff with jitter.
How it connects
Connects retry storms to the Bullwhip Effect, latency, circuit breakers, exponential backoff, jitter, and cascading failure.

Also explained in

Compare how 2 books explain retry storm →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Software Architecture. The section links point to the parts of the book it draws on.