liveBook · Manning

Retry

As explained in Designing AI Systems

A retry is a repeated attempt to perform an operation after a transient failure or unsuccessful attempt.

A retry repeats an operation after a failure that may be temporary. Network timeouts, rate limits, and temporary service unavailability can justify another attempt; an invalid request generally cannot. A retry policy limits attempts and delay so recovery does not turn into repeated pressure on a struggling service. section 3.4.1

A practical retry policy specifies:

  • Which failures qualify: for example, HTTP 429, timeouts, and HTTP 503, rather than HTTP 400. section 3.4.1
  • A retry limit and an initial delay, with optional exponential backoff and a maximum delay. Backoff can wait one, two, and then four seconds between successive retries. section 3.4.1
  • What happens afterward: fail, let the caller handle the error, or try an ordered fallback provider. Some errors can be configured to fail without fallback. section 9.3.2 section 3.4.2
Listing 3.11 RetryConfig dataclass for transient failure handling
@dataclass
class RetryConfig:
    max_retries: int = 3    #A
    initial_delay: float = 1.0
    exponential_backoff: bool = True    #B
    max_delay: float = 60.0
    retry_on: List[FailureType] = None    #C
 

RetryConfig groups the attempt limit, delay controls, and eligible failure types into a reusable model-service policy. section 3.4.1

Retry at the boundary where the failure occurs. A workflow SDK interceptor retries transient gRPC failures between a workflow container and platform services; the Model Service independently retries failures between itself and external providers. The application configures the policies while the platform executes them. section 9.3.2 section 3.4

Listing 9.13 gRPC retry interceptor for SDK service calls
import time
import grpc
 
RETRYABLE_CODES = {
    grpc.StatusCode.UNAVAILABLE,    #A
    grpc.StatusCode.DEADLINE_EXCEEDED,    #A
    grpc.StatusCode.RESOURCE_EXHAUSTED,    #A
}
 
class RetryInterceptor(
    grpc.UnaryUnaryClientInterceptor
):
    def __init__(self, max_retries=3,
                 initial_delay=0.1):
        self._max_retries = max_retries    #B
        self._initial_delay = initial_delay
 
    def intercept_unary_unary(
        self, continuation, call_details,
        request
    ):
        last_error = None
        for attempt in range(    #C
            self._max_retries + 1    #C
        ):    #C
            try:
                return continuation(    #D
                    call_details, request    #D
                )    #D
            except grpc.RpcError as e:
                last_error = e
                if (e.code() not in    #E
                        RETRYABLE_CODES):    #E
                    raise    #E
                if attempt == self._max_retries:
                    raise    #F
                delay = (self._initial_delay    #G
                    * (2 ** attempt))    #G
                time.sleep(delay)    #G
        raise last_error

The gRPC interceptor retries UNAVAILABLE, DEADLINE_EXCEEDED, and RESOURCE_EXHAUSTED; other errors propagate immediately. section 9.3.2

For workflow-to-workflow HTTP calls, the retry loop treats server errors and connection failures as transient, but returns client errors without retrying. Its loop permits an initial request plus max_retries additional attempts, then raises the last error. section 9.4.3

Choose a policy for the workload

A patient-facing assistant can use short delays and a fallback chain to restore service quickly. A batch analytics pipeline can use longer backoff without fallback, leaving failed items for the pipeline to skip, queue, or alert on. A synchronous workflow with a tight timeout needs fewer, shorter retries than a long-running asynchronous workflow. section 3.4.3 section 9.1.1

Keep retries observable even when they hide failures from callers. Frequent fallback use can indicate a struggling primary provider; workflow metrics can reveal which steps fail and how long they take. Shallow readiness checks can leave transient dependency failures to per-request retries rather than removing healthy containers from service. section 3.4.2 section 2.1.7 section 9.3.3

How Designing AI Systems covers it

How deep it goes
Defines retry policy fields and error classification, then implements exponential-backoff loops for gRPC and HTTP calls.
Examples
Contrasts a patient-facing assistant's quick failover with a batch analytics pipeline's longer retries and disabled fallback.
What you can do after
Operate/configure retry limits, delays, eligible errors, and provider fallbacks for different availability and latency needs.
How it connects
Connects retries with exponential backoff, fallback, timeouts, and health checks.

Also explained in

Compare how 25 books explain retry →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.