liveBook · Manning

Version

As explained in AI Model Evaluation

A model version is a distinct released or identified state of a model with a specific implementation, parameters, or training artifacts.

A model version identifies a distinct released or evaluated state whose implementation, parameters, prompt configuration, or training artifacts may differ. Recording the exact version turns evaluation results into traceable evidence: teams can distinguish a model change from changed data, prompts, judges, or schemas. section 9.10.4 section 10.6.2

Treat version as an operational record, not merely a label:

  • Log the model or evaluator version alongside prompts, timestamps, response schemas, outputs, token usage, and evaluation results. section 9.10.4
  • Pin identifiers such as gpt-4o-2024-05-13 and retain output hashes so later runs can be compared for consistency. section 9.10.4
  • Use fixed canary prompts to detect silent changes in evaluator behavior before treating score movement as model regression. section 9.10.4 section 10.6.2
  • During triage, check data, judge version, rubric and prompt, stability, and only then the evaluated model version. section 10.6.2

Ranking-policy version in a search log

A search event records the query, context, candidate set, selected ranking, propensities, downstream outcomes, and the policy version that made the decision. That version lets counterfactual evaluation connect later clicks or purchases to the precise ranking state that produced them. section 5.5.5

Listing 5.3 Search log event example for a ecommerce website
{
  "session_id": "sess_20241125_210315_user_78234",
  "timestamp": "2024-11-25T21:03:15.342Z",
  "user_id": "user_78234",
  "context": {
    "page": "search_results",
    "device": "desktop",
    "query": "wireless earbuds",
    "time_of_day": "evening"
  },
  "available_actions": [
    {
      "action_id": "rank_position_1",
      "product_id": "prod_84729",
      "product_name": "AirPods Pro Max",
      "price": 249.99,
      "avg_rating": 4.6,
      "relevance_score": 0.94
    },
    {
      "action_id": "rank_position_1",
      "product_id": "prod_52846", 
      "product_name": "Sony WF-1000XM4",
      "price": 199.99,
      "avg_rating": 4.4,
      "relevance_score": 0.91
    },
    {
      "action_id": "rank_position_1",
      "product_id": "prod_73958",
      "product_name": "Bose QuietComfort Earbuds",
      "price": 179.99,
      "avg_rating": 4.2,
      "relevance_score": 0.88
    },
    {
      "action_id": "rank_position_1",
      "product_id": "prod_91647",
      "product_name": "Samsung Galaxy Buds Pro",
      "price": 149.99,
      "avg_rating": 4.3,
      "relevance_score": 0.85
    }
  ],
  "action_probabilities": {
    "prod_84729": 0.52,
    "prod_52846": 0.28,
    "prod_73958": 0.13,
    "prod_91647": 0.07
  },
  "selected_action": {
    "product_id": "prod_84729",
    "selection_probability": 0.52,
    "model_version": "search_ranking_v3.1.8"
  },
  "outcome": {
    "clicked": true,
    "click_position": 1,
    "time_to_click_seconds": 6.8,
    "added_to_cart": false,
    "viewed_alternatives": ["prod_52846", "prod_73958"]
  }
}

Inspect a complete search event showing how a versioned ranking decision sits alongside context, available actions, scores, and outcomes. section 5.5.5

How AI Model Evaluation covers it

How deep it goes
Operationalizes versioning through identifiers, hashes, canaries, logs, and triage, without descending into model implementation internals.
Examples
Uses dated evaluator identifiers, output hashes, fixed canary prompts, and a search event containing the ranking policy version.
What it assumes
Assumes an evaluation pipeline that records prompts, data, outputs, results, and model or policy identifiers.
What you can do after
Diagnose evaluation changes by separating data, judge, prompt, instability, and evaluated-model causes.
How it connects
Connects model versions with observability, prompt versioning, canary testing, counterfactual evaluation, and regression diagnosis.

Also explained in

Compare how 7 books explain version →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.