A model version identifies a distinct released or evaluated state whose implementation, parameters, prompt configuration, or training artifacts may differ. Recording the exact version turns evaluation results into traceable evidence: teams can distinguish a model change from changed data, prompts, judges, or schemas. section 9.10.4 section 10.6.2
Treat version as an operational record, not merely a label:
- Log the model or evaluator version alongside prompts, timestamps, response schemas, outputs, token usage, and evaluation results. section 9.10.4
- Pin identifiers such as gpt-4o-2024-05-13 and retain output hashes so later runs can be compared for consistency. section 9.10.4
- Use fixed canary prompts to detect silent changes in evaluator behavior before treating score movement as model regression. section 9.10.4 section 10.6.2
- During triage, check data, judge version, rubric and prompt, stability, and only then the evaluated model version. section 10.6.2
Ranking-policy version in a search log
A search event records the query, context, candidate set, selected ranking, propensities, downstream outcomes, and the policy version that made the decision. That version lets counterfactual evaluation connect later clicks or purchases to the precise ranking state that produced them. section 5.5.5
Listing 5.3 Search log event example for a ecommerce website
{
"session_id": "sess_20241125_210315_user_78234",
"timestamp": "2024-11-25T21:03:15.342Z",
"user_id": "user_78234",
"context": {
"page": "search_results",
"device": "desktop",
"query": "wireless earbuds",
"time_of_day": "evening"
},
"available_actions": [
{
"action_id": "rank_position_1",
"product_id": "prod_84729",
"product_name": "AirPods Pro Max",
"price": 249.99,
"avg_rating": 4.6,
"relevance_score": 0.94
},
{
"action_id": "rank_position_1",
"product_id": "prod_52846",
"product_name": "Sony WF-1000XM4",
"price": 199.99,
"avg_rating": 4.4,
"relevance_score": 0.91
},
{
"action_id": "rank_position_1",
"product_id": "prod_73958",
"product_name": "Bose QuietComfort Earbuds",
"price": 179.99,
"avg_rating": 4.2,
"relevance_score": 0.88
},
{
"action_id": "rank_position_1",
"product_id": "prod_91647",
"product_name": "Samsung Galaxy Buds Pro",
"price": 149.99,
"avg_rating": 4.3,
"relevance_score": 0.85
}
],
"action_probabilities": {
"prod_84729": 0.52,
"prod_52846": 0.28,
"prod_73958": 0.13,
"prod_91647": 0.07
},
"selected_action": {
"product_id": "prod_84729",
"selection_probability": 0.52,
"model_version": "search_ranking_v3.1.8"
},
"outcome": {
"clicked": true,
"click_position": 1,
"time_to_click_seconds": 6.8,
"added_to_cart": false,
"viewed_alternatives": ["prod_52846", "prod_73958"]
}
}
Inspect a complete search event showing how a versioned ranking decision sits alongside context, available actions, scores, and outcomes. section 5.5.5






