A retry storm is a systemic overload condition in which multiple services repeatedly retry failed requests, amplifying traffic and load. Retries can recover from transient failures, but simultaneous attempts without coordination or delay can overwhelm an already struggling service and turn a localized fault into an outage. section 12.3.2
Investigate these warning signs:
- Elevated CPU, request volume, or latency without increased user traffic. section 12.3.2
- Repeated log entries with identical or similar payloads and retry attempts. section 12.3.2
- Short repeated spans, duplicate request IDs, or a sudden traffic spike at a downstream service. section 12.4
- A chain reaction in which cascading failures, exhausted resources, or timeout mismatches increase retry pressure. section 12.3.2
Correlate CPU, request rate, and error rate across services with metrics dashboards. Follow trace IDs and spans to locate repeated calls, compare logs with traffic changes, and check for chain reactions rather than treating the issue as a single local exception. section 12.3.2 section 12.4
Repeated payment retries
A checkout service logs timeout retries for the same PaymentService request: attempt 1, attempt 2, and repeated POST operations for orderId=abc123. A trace showing three payment attempts within one checkout makes the amplification visible. section 12.3.2
