Model selection is choosing a model for an application before tuning parameters or prompts. The choice sets the cost ceiling, latency floor, capability range, and available parameters. section 2.1
Treat selection as both an initial design decision and ongoing operations:
- Keep the model name in configuration rather than hardcoding it, so you can replace a model when options or pricing change, or when it is deprecated. section 2.1.3
- Reassess fit using shadow tests and quality monitoring. Consider switching when a cheaper candidate performs comparably, traffic patterns change, or capability gaps become important. section 10.5.3
- For fine-tuning, choose both a base model and a fine-tuning method as part of the workflow between data preparation and training. section 5.3
async def handle_query_with_shadow(question):
production_response = production_model.generate(question) #1
asyncio.create_task( #2
shadow_test(question, production_response, candidate_model)
)
return production_response #1
async def shadow_test(question, production_response, #3
candidate_model): #3
candidate_response = candidate_model.generate(question) #3
#3
log_shadow_result({ #3
"question": question, #3
"production_response": production_response, #3
"candidate_response": candidate_response, #3
"timestamp": datetime.now() #3
})
Shadow-test a candidate alongside the production model, log both responses, and continue returning the production response. section 10.5.3
