liveBook · Manning

Retry

As explained in Grokking Software Architecture

A retry is a fault-tolerance technique that repeats a failed or apparently lost operation according to a defined policy.

A payment request fails, but was the service down—or did a network packet simply go missing? A retry gives a failed or apparently lost operation another chance. It can rescue a transaction from a brief glitch, but an unchecked retry can make a struggling service even sicker. section 10.2.2 section 10.4.1

A safer retry policy makes a few important choices:

  • Retry temporary failures, such as network blips, socket timeouts, or HTTP 503 responses. Fail fast on permanent errors such as invalid requests or authentication failures; route corrupted messages to a dead letter queue. section 10.1
  • Space attempts with exponential backoff. After successive failures, wait progressively longer—for example, 1, 2, 4, then 8 seconds—instead of hammering the service immediately. section 10.2.2
  • Add random jitter to the delays and use a circuit breaker to fail fast when a downstream service is struggling. Together, these give it breathing room and help prevent cascading failures. section 12.8.3
  • Protect operations with side effects. For a payment, generate an idempotency key in the business layer and pass the same key with every attempt, so each retry carries the same identity. section 10.2.2

A tiny delay becomes a retry storm

A gateway calls an order service, which calls a payment API. A two-second database delay can make the order service assume a request was lost. With 1,000 users and a naive loop allowing three attempts, the payment API can receive 3,000 unspaced requests—turning a delay into a much bigger outage. section 12.8.3

Listing 12.3 The Retry Storm
public async Task ProcessPaymentAsync(Order order)
{
    int retryCount = 0;
    bool success = false;
 
    while (!success && retryCount < 3) #A
    {
        try
        {
            var response = await _httpClient.PostAsync(
                "http://payment-api/charge", order);
            response.EnsureSuccessStatusCode();
            success = true;
        }
        catch (HttpRequestException)
        {
            retryCount++; #B
            _logger.LogWarning(
                $"Payment timeout. Firing retry attempt {retryCount}...");
        }
    }
}

The payment loop caps attempts at three but sends them without a delay, illustrating how even bounded retries can amplify traffic. section 12.8.3

Retries also appear in message processing. If a worker finishes a job but crashes before acknowledging it, the broker can put the message back on the queue. Another worker may repeat the job, so idempotency matters here too. section 7.4.3

A retry is not a substitute for a timeout or a fallback. A timeout stops an external call from waiting indefinitely; a fallback provides another way forward when the dependency remains unavailable. section 10.10 section 10.2

How Grokking Software Architecture covers it

How deep it goes
Explains failure classification, exponential backoff, jitter, idempotency, and retry storms through concrete scenarios and a retry loop.
Examples
Uses a payment API retry storm and a message resent after a missing acknowledgment to illustrate overload and duplicate processing.
What you can do after
Evaluate/understand which failures warrant retries and how to reduce overload and duplicate-operation risks.
How it connects
Connects retries to timeouts, circuit breakers, fallbacks, idempotency, and at-least-once delivery.

Also explained in

Compare how 25 books explain retry →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Software Architecture. The section links point to the parts of the book it draws on.