liveBook · Manning

Fallback configuration

As explained in Designing AI Systems

Fallback configuration defines alternative providers, their retry settings, and failure conditions for execution after an earlier provider path is unavailable or exhausted.

Fallback configuration defines ordered alternative providers for execution after an earlier provider path exhausts its retries or becomes unavailable. Each fallback can use its own retry settings, while selected failures bypass fallback entirely. section 3.10 section 3.4.2

A production fallback policy specifies:

  • An enabled flag and an ordered providers list; earlier entries receive preference. section 3.4.2
  • A retry_config for each provider before the system advances to the next provider. section 3.10 section 3.4.2
  • A fail_on list for errors such as malformed requests or content-policy violations that should fail immediately. section 3.4.2
  • Operational logging of fallback events and the provider that ultimately handled each successful request. section 3.4.2

Provider failover configuration

A request may try GPT-4 first, then Claude, then a local Llama model. A corresponding configuration can be represented as: @dataclass class FallbackConfig: enabled: bool = True; providers: List[str] = None; retry_config: RetryConfig = None; fail_on: List[FailureType] = None. If Claude succeeds after GPT-4 fails, the response identifies Claude for logging, cost attribution, and behavioral analysis. section 3.4.2

How Designing AI Systems covers it

How deep it goes
Defines configuration fields, ordering, error exceptions, and operational logging, but does not detail fallback execution internals.
Examples
Uses GPT-4, Claude, and a local Llama model in an ordered fallback chain.
What it assumes
Assumes provider requests and retry behavior, while teaching how exhausted retries transition into alternate providers.
What you can do after
Operate/configure ordered provider failover with per-provider retries, immediate-failure rules, and provider attribution.
How it connects
Connects retries, provider routing, failure handling, logging, and cost attribution.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.