Shadow testing runs a candidate model alongside the production model on real queries. Candidate responses are logged for comparison but never shown to users, allowing evaluation without changing the live experience. section 10.5.3
A production-minded setup should:
- Run candidate requests asynchronously so they do not add latency to user requests. section 10.5.3
- Budget for roughly doubled model costs during evaluation, and limit tests to representative time windows. section 10.5.3
async def handle_query_with_shadow(question):
production_response = production_model.generate(question) #1
asyncio.create_task( #2
shadow_test(question, production_response, candidate_model)
)
return production_response #1
async def shadow_test(question, production_response, #3
candidate_model): #3
candidate_response = candidate_model.generate(question) #3
#3
log_shadow_result({ #3
"question": question, #3
"production_response": production_response, #3
"candidate_response": candidate_response, #3
"timestamp": datetime.now() #3
})
The Python sketch returns the production response immediately, then asynchronously runs the candidate model and logs both responses with a timestamp. section 10.5.3
