liveBook · Manning

Candidate set

As explained in AI Model Evaluation

A candidate set is the collection of items or actions considered for selection by a decision-making system.

A candidate set is the collection of feasible items or actions available when a decision-making system chooses an outcome. It captures what the system could reasonably have selected, not only what it actually selected. In counterfactual evaluation, preserving this set makes alternative decisions observable after the fact. section 5.3 section 5.5.1

A useful candidate-set record preserves the decision context and the evidence needed to compare alternatives:

  • Record the candidates considered at decision time, including alternatives that were not selected. section 5.3 section 5.5.1
  • Associate candidates with scores or selection probabilities when the evaluation estimator needs them. section 5.5.1 section 5.5.5
  • Log the selected action, context, policy version, and downstream outcome alongside the candidate set. section 5.3 section 5.5.5
  • Monitor missing fields, schema changes, and delayed outcome joins because incomplete records weaken evaluation reliability. section 5.5.5

Search ranking

For a search for wireless earbuds, a ranker might consider AirPods Pro Max, Sony WF-1000XM4, and Bose QuietComfort Earbuds for the first position. Logging all candidates, their scores or probabilities, the selected ranking, query context, and later clicks or purchases supports questions such as what might have happened if Sony earbuds had ranked first. section 5.5.1 section 5.5.5

Listing 5.3 Search log event example for a ecommerce website
{
  "session_id": "sess_20241125_210315_user_78234",
  "timestamp": "2024-11-25T21:03:15.342Z",
  "user_id": "user_78234",
  "context": {
    "page": "search_results",
    "device": "desktop",
    "query": "wireless earbuds",
    "time_of_day": "evening"
  },
  "available_actions": [
    {
      "action_id": "rank_position_1",
      "product_id": "prod_84729",
      "product_name": "AirPods Pro Max",
      "price": 249.99,
      "avg_rating": 4.6,
      "relevance_score": 0.94
    },
    {
      "action_id": "rank_position_1",
      "product_id": "prod_52846", 
      "product_name": "Sony WF-1000XM4",
      "price": 199.99,
      "avg_rating": 4.4,
      "relevance_score": 0.91
    },
    {
      "action_id": "rank_position_1",
      "product_id": "prod_73958",
      "product_name": "Bose QuietComfort Earbuds",
      "price": 179.99,
      "avg_rating": 4.2,
      "relevance_score": 0.88
    },
    {
      "action_id": "rank_position_1",
      "product_id": "prod_91647",
      "product_name": "Samsung Galaxy Buds Pro",
      "price": 149.99,
      "avg_rating": 4.3,
      "relevance_score": 0.85
    }
  ],
  "action_probabilities": {
    "prod_84729": 0.52,
    "prod_52846": 0.28,
    "prod_73958": 0.13,
    "prod_91647": 0.07
  },
  "selected_action": {
    "product_id": "prod_84729",
    "selection_probability": 0.52,
    "model_version": "search_ranking_v3.1.8"
  },
  "outcome": {
    "clicked": true,
    "click_position": 1,
    "time_to_click_seconds": 6.8,
    "added_to_cart": false,
    "viewed_alternatives": ["prod_52846", "prod_73958"]
  }
}

Inspect a structured search log showing context, available actions, product attributes, and probabilities for a candidate set. section 5.5.5

How AI Model Evaluation covers it

How deep it goes
Explains candidate-set contents, associated probabilities, contextual fields, and logging failures, but stops short of estimator mathematics.
Examples
Uses recommendation alternatives and a wireless-earbuds search event with product candidates, scores, probabilities, and engagement outcomes.
What you can do after
Configure decision logs that preserve alternatives and evaluate what might have happened under another ranking or selection policy.
How it connects
Connects candidate sets to counterfactual evaluation, propensity scores, recommendation, search ranking, policy versions, and downstream outcomes.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.