Dynamic routing selects a model for each input or combines outputs from several models. In generative-AI applications, it lets specialized models handle the requests they serve best. section 10.5.1
An implementation typically considers:
- Model-selection criteria such as performance, training-data diversity, and specialization. section 10.5.1
- API integration across endpoints, including throttling, rate limits, error handling, and response caching. section 10.5.1
- Scalability measures such as load balancing, API gateways, regional redundancy, queues, and stream processing. section 10.5.1
Intent-based model selection
A fast, locally running Phi-2 classifier checks whether a question concerns dogs. Relevant questions can continue through the local flow; irrelevant questions are routed to GPT-4. A separate classifier pattern maps enterprise questions to numbered approaches.
