liveBook · Manning

Exponential backoff

As explained in Designing AI Systems

Exponential backoff is a retry timing strategy that increases the delay between successive attempts, typically by a multiplicative factor.

Exponential backoff spaces retry attempts with progressively longer delays, commonly multiplying each wait: one second, then two, then four. A maximum delay caps the growth, preventing recovery attempts from becoming unreasonably slow while reducing pressure on an impaired service. section 3.4.1

Use backoff as part of an explicit retry policy:

  • Retry transient failures such as timeouts, rate limits, service unavailability, gRPC UNAVAILABLE, DEADLINE_EXCEEDED, and RESOURCE_EXHAUSTED. section 3.4.1 section 9.3.2
  • Reject permanent failures such as invalid requests immediately instead of repeating the same bad operation. section 3.4.1 section 9.3.2
  • Keep retry layers distinct: SDK clients recover platform-network failures, while the Model Service recovers external-provider failures. section 9.3.2

Provider retry configuration

A production-oriented configuration can allow three retries, start at one second, enable exponential growth, cap waits at 60 seconds, and retry only selected failure types. Shorter retries can support fast failover; longer retries suit batch pipelines that handle failed items separately. section 3.4.1 section 3.4.3

Listing 3.11 RetryConfig dataclass for transient failure handling
@dataclass
class RetryConfig:
    max_retries: int = 3    #A
    initial_delay: float = 1.0
    exponential_backoff: bool = True    #B
    max_delay: float = 60.0
    retry_on: List[FailureType] = None    #C
 

Inspect the RetryConfig dataclass to see the policy fields for attempt limits, initial delay, exponential growth, maximum delay, and retryable failure types. section 3.4.1

How Designing AI Systems covers it

How deep it goes
Defines configuration fields, failure selection, delay growth, caps, and layered interceptors, but does not detail jitter.
Examples
Uses provider configurations, batch pipelines, fallback chains, and automatic gRPC SDK retries as concrete examples.
What it assumes
Assumes familiarity with transient versus permanent errors, provider fallbacks, and network service boundaries.
What you can do after
Configure retry behavior with attempt limits, delays, caps, and failure filters in provider services and gRPC clients.
How it connects
Connects retry policy with transient failures, rate limiting, fallbacks, gRPC interceptors, and service boundaries.

Also explained in

Compare how 11 books explain exponential backoff →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.