Hybrid deployment combines local models with hosted APIs to balance capability, cost, and control. A common pattern handles routine queries locally and sends harder cases to a hosted model, with fallback for reliability. section 10.2.3 section 10.7
A production router can:
- Send short factual questions to a local model and route nuanced or risky queries to a hosted model. section 10.2.3
- Fall back to a hosted API when local models fail or are overloaded; shift traffic using cost and latency. section 10.2.3
- Avoid relying on keyword matching alone; use semantic routing or a fast, inexpensive LLM to classify intent. section 10.2.3
Figure 10.2 Hybrid LLM architecture combining local and hosted models for optimal cost, performance, and control. The smart router analyzes query characteristics to direct simple questions to cost-effective local models while routing complex or creative tasks to more capable hosted APIs, with automatic fallback ensuring reliability.
Diagram of a smart router directing simple queries to local models and complex tasks to hosted APIs, with automatic fallback. section 10.2.3
