liveBook · Manning

Fallback

As explained in Designing AI Systems

A fallback is a failure-handling strategy that switches to an alternative provider or execution path when the primary one fails.

A fallback switches a failed request to an alternative provider or model. In a model service, it keeps application code independent of provider-specific failure handling: the application configures the policy, and the service executes it. section 3.4 section 3.0

Fallback handles failure after a request executes; routing selects a provider before execution. Retries offer another recovery path by attempting an eligible failure again. Errors outside the retry list can proceed directly to fallback, while policy can also specify errors that should fail. section 3.5 section 3.4.1 section 3.4.3

Choose the policy around the application's operational needs:

  • Prioritize availability with short retries and a chain of cloud and local alternatives. A local last resort can also keep sensitive data on-premises if cloud providers fail. section 3.4.3
  • Disable fallbacks for a batch pipeline when failures should surface for pipeline-level handling; use longer retries with exponential backoff for transient errors. section 3.4.3
  • Move rate-limited requests to cheaper alternatives instead of waiting, accepting potentially lower response quality during high traffic. section 3.4.3
FallbackConfig(
    enabled=True,
    providers=["claude-sonnet-4-5", "llama-local"],
    retry_config=RetryConfig(
        max_retries=2,
        initial_delay=0.5,
        retry_on=[FailureType.RATE_LIMIT, FailureType.TIMEOUT, 
➥FailureType.SERVICE_UNAVAILABLE]
    )
)
 

This configuration combines a two-provider fallback chain with short retries for rate limits, timeouts, and service unavailability. section 3.4.3

Streaming constrains recovery. Once tokens have reached the client, switching providers mid-response can produce a jarring change in style. Most configurations therefore fall back only before streaming begins. A mid-stream failure instead ends with an error chunk, leaving the partial response available to display. section 3.3.3

Record the provider and model that actually served each request, not only the ones requested. Track time spent in retries or fallback attempts and classify failures so operators can attribute cost and investigate recurring problems. section 3.8.1

How Designing AI Systems covers it

How deep it goes
Explains failure-specific retry and fallback configurations, operational tradeoffs, and the limits of recovery during streaming.
Examples
Contrasts a patient-facing cloud-to-local chain, a batch pipeline without fallbacks, and cheaper alternatives for rate-limited requests.
What you can do after
Operate and configure fallback chains and retry policies for availability, batch processing, or cost-sensitive workloads.
How it connects
Connects retries, model routing, streaming, and observability to provider failure handling.

Also explained in

Compare how 24 books explain fallback →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.