Sending every customer-support query to a powerful model can make routine FAQs expensive and slow. Sending everything to a cheaper model risks underserving questions that need detailed reasoning. Multimodel fallback uses a router to match each request to a model: fast and inexpensive for straightforward questions, more capable for complex or specialized ones. section 9.2.4
A practical routing path is:
- Inspect the query for signals of complexity. A simple Python heuristic checks for terms such as “legal” or “tax” and treats queries longer than 30 words as complex; a model-based classifier can instead return a structured simple-or-complex decision. section 9.2.4
- Send routine requests to the cheaper model and escalate likely complex requests to the advanced model. This is a decision made before the answer, not a demonstrated retry after checking whether an answer was adequate. section 9.2.4
- Combine routing with other service-level choices: batching handles bursts, semantic caching can reuse answers, and streaming controls how a selected model’s response reaches the user. section 9.2 section 9.2.5
Holiday-sale support
An online store can route “What time do you open?” to gpt-5.4-mini while reserving gpt-5.4 for a request about legal disclaimers for international returns. In the Python service, each request is queued; its handler chooses a model, then either streams a response or retrieves a final answer through the caching path. The examples show how to wire the decision into request handling, but keyword and length checks remain rough proxies for difficulty, not guarantees of answer quality. section 9.2.4 section 9.2.5
async def handle_request(req: dict):
"""
Extracts the query, picks the model,
decides if we stream or just retrieve a final answer,
then calls the request's callback with the result.
"""
query = req["query"]
streaming = req["streaming"]
cb = req["callback"]
model = choose_model(query)
if streaming:
print(f"\n[Streaming Mode] '{query}' => Using {model}")
await stream_response(query, model)
cb("[Stream Complete]")
else:
answer = get_cached_answer(query, model)
cb(answer)
The request handler makes the routing point explicit: choose a model from the query before taking the streaming or cached-answer path. section 9.2.5
The goal is to spend advanced-model capacity where it matters without making every user wait for it. Treat routing rules as an operational choice to assess against both cost and response quality: a cheaper answer is not a success if the request needed deeper reasoning. section 9.2.4 section 9.4
