A payment request fails, but was the service down—or did a network packet simply go missing? A retry gives a failed or apparently lost operation another chance. It can rescue a transaction from a brief glitch, but an unchecked retry can make a struggling service even sicker. section 10.2.2 section 10.4.1
A safer retry policy makes a few important choices:
- Retry temporary failures, such as network blips, socket timeouts, or HTTP 503 responses. Fail fast on permanent errors such as invalid requests or authentication failures; route corrupted messages to a dead letter queue. section 10.1
- Space attempts with exponential backoff. After successive failures, wait progressively longer—for example, 1, 2, 4, then 8 seconds—instead of hammering the service immediately. section 10.2.2
- Add random jitter to the delays and use a circuit breaker to fail fast when a downstream service is struggling. Together, these give it breathing room and help prevent cascading failures. section 12.8.3
- Protect operations with side effects. For a payment, generate an idempotency key in the business layer and pass the same key with every attempt, so each retry carries the same identity. section 10.2.2
A tiny delay becomes a retry storm
A gateway calls an order service, which calls a payment API. A two-second database delay can make the order service assume a request was lost. With 1,000 users and a naive loop allowing three attempts, the payment API can receive 3,000 unspaced requests—turning a delay into a much bigger outage. section 12.8.3
Listing 12.3 The Retry Storm
public async Task ProcessPaymentAsync(Order order)
{
int retryCount = 0;
bool success = false;
while (!success && retryCount < 3) #A
{
try
{
var response = await _httpClient.PostAsync(
"http://payment-api/charge", order);
response.EnsureSuccessStatusCode();
success = true;
}
catch (HttpRequestException)
{
retryCount++; #B
_logger.LogWarning(
$"Payment timeout. Firing retry attempt {retryCount}...");
}
}
}The payment loop caps attempts at three but sends them without a delay, illustrating how even bounded retries can amplify traffic. section 12.8.3
Retries also appear in message processing. If a worker finishes a job but crashes before acknowledging it, the broker can put the message back on the queue. Another worker may repeat the job, so idempotency matters here too. section 7.4.3
A retry is not a substitute for a timeout or a fallback. A timeout stops an external call from waiting indefinitely; a fallback provides another way forward when the dependency remains unavailable. section 10.10 section 10.2
























