liveBook · Manning

Routing

As explained in Designing AI Systems

Routing is the selection of a model provider for a request according to a configured strategy and runtime conditions.

Routing selects a model provider before a request executes. It lets applications use a common model interface while the Model Service chooses a provider based on configured preferences and runtime conditions. Unlike routing, fallback handles failures after execution begins. section 3.5 section 1.3

Applications configure routing preferences; the Model Service executes them and maintains shared state such as spending, active request counts, and provider capabilities. Built-in strategies can be combined or replaced with custom logic. An explicit provider bypasses routing; otherwise, a default provider can resolve cases where the strategy produces no clear winner. section 3.5 section 3.5.1

Listing 3.13 RoutingConfig dataclass for provider selection
@dataclass
class RoutingConfig:
    strategy: RoutingStrategy    #A
    default_provider: str = None    #B
    cost_config: CostRoutingConfig = None
    load_config: LoadRoutingConfig = None
    feature_config: FeatureRoutingConfig = None
 

RoutingConfig selects a strategy and default provider, with separate settings for cost, load, and feature routing. section 3.5.1

Common strategies address different selection constraints:

  • Cost-aware routing considers spending and task complexity, reserving more expensive models for work that justifies them. The service calculates running spending from response token counts and configured provider prices. section 3.5.2
  • Load-based routing distributes requests across providers using information such as active request counts. section 3.10 section 3.5
  • Feature-based routing matches requests to providers with the required capabilities. section 3.10

Route against a monthly budget

A healthcare application sets a $5,000 monthly budget. Routine patient questions go to inexpensive models, while complex medical reasoning uses more powerful models. Near the budget limit, requests shift to a local model to preserve funds for emergencies. section 3.5.2

In the ModelClient chat interface, an empty model name lets the Model Service apply its routing logic. The response identifies the model that generated it, regardless of the requested model. section 3.9.2

How Designing AI Systems covers it

How deep it goes
Explains configuration, shared routing state, and strategy choices, with a worked cost-aware decision flow.
Examples
Illustrates routing with a healthcare application allocating a $5,000 monthly budget across routine questions, complex reasoning, and local models.
What you can do after
Configure routing strategies and a default provider, or bypass routing by specifying a provider explicitly.
How it connects
Connects provider selection to model abstraction, fallback, load balancing, and capability matching.

Also explained in

Compare how 34 books explain routing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.