A semantic router reads a prompt, predicts which language model best fits the request, then sends the request to that model and processes its output. This makes a collection of smaller or specialized models act as one system. section 10.1.2
A practical routing workflow can:
- Break a prompt into the steps required for the task. section 10.1.2
- Generate API queries that reach into external systems when the task requires them. section 10.1.2
- Choose an appropriate LLM or downstream processing path from the prompt's meaning. section 10.1.2
- Use natural-language classification to distinguish, for example, knowledge-base questions from other queries. section 10.1.3
Routing a chatbot request
A chatbot can classify an incoming request as a question and send it to its knowledge base, while routing other requests elsewhere. A few training examples can support generalization, potentially reducing engineering and compute costs compared with sending every request through a larger intermediary model. section 10.1.3
