Routing selects a model provider before a request executes. It lets applications use a common model interface while the Model Service chooses a provider based on configured preferences and runtime conditions. Unlike routing, fallback handles failures after execution begins. section 3.5 section 1.3
Applications configure routing preferences; the Model Service executes them and maintains shared state such as spending, active request counts, and provider capabilities. Built-in strategies can be combined or replaced with custom logic. An explicit provider bypasses routing; otherwise, a default provider can resolve cases where the strategy produces no clear winner. section 3.5 section 3.5.1
Listing 3.13 RoutingConfig dataclass for provider selection
@dataclass
class RoutingConfig:
strategy: RoutingStrategy #A
default_provider: str = None #B
cost_config: CostRoutingConfig = None
load_config: LoadRoutingConfig = None
feature_config: FeatureRoutingConfig = None
RoutingConfig selects a strategy and default provider, with separate settings for cost, load, and feature routing. section 3.5.1
Common strategies address different selection constraints:
- Cost-aware routing considers spending and task complexity, reserving more expensive models for work that justifies them. The service calculates running spending from response token counts and configured provider prices. section 3.5.2
- Load-based routing distributes requests across providers using information such as active request counts. section 3.10 section 3.5
- Feature-based routing matches requests to providers with the required capabilities. section 3.10
Route against a monthly budget
A healthcare application sets a $5,000 monthly budget. Routine patient questions go to inexpensive models, while complex medical reasoning uses more powerful models. Near the budget limit, requests shift to a local model to preserve funds for emergencies. section 3.5.2
In the ModelClient chat interface, an empty model name lets the Model Service apply its routing logic. The response identifies the model that generated it, regardless of the requested model. section 3.9.2

































