A candidate set is the collection of feasible items or actions available when a decision-making system chooses an outcome. It captures what the system could reasonably have selected, not only what it actually selected. In counterfactual evaluation, preserving this set makes alternative decisions observable after the fact. section 5.3 section 5.5.1
A useful candidate-set record preserves the decision context and the evidence needed to compare alternatives:
- Record the candidates considered at decision time, including alternatives that were not selected. section 5.3 section 5.5.1
- Associate candidates with scores or selection probabilities when the evaluation estimator needs them. section 5.5.1 section 5.5.5
- Log the selected action, context, policy version, and downstream outcome alongside the candidate set. section 5.3 section 5.5.5
- Monitor missing fields, schema changes, and delayed outcome joins because incomplete records weaken evaluation reliability. section 5.5.5
Search ranking
For a search for wireless earbuds, a ranker might consider AirPods Pro Max, Sony WF-1000XM4, and Bose QuietComfort Earbuds for the first position. Logging all candidates, their scores or probabilities, the selected ranking, query context, and later clicks or purchases supports questions such as what might have happened if Sony earbuds had ranked first. section 5.5.1 section 5.5.5
Listing 5.3 Search log event example for a ecommerce website
{
"session_id": "sess_20241125_210315_user_78234",
"timestamp": "2024-11-25T21:03:15.342Z",
"user_id": "user_78234",
"context": {
"page": "search_results",
"device": "desktop",
"query": "wireless earbuds",
"time_of_day": "evening"
},
"available_actions": [
{
"action_id": "rank_position_1",
"product_id": "prod_84729",
"product_name": "AirPods Pro Max",
"price": 249.99,
"avg_rating": 4.6,
"relevance_score": 0.94
},
{
"action_id": "rank_position_1",
"product_id": "prod_52846",
"product_name": "Sony WF-1000XM4",
"price": 199.99,
"avg_rating": 4.4,
"relevance_score": 0.91
},
{
"action_id": "rank_position_1",
"product_id": "prod_73958",
"product_name": "Bose QuietComfort Earbuds",
"price": 179.99,
"avg_rating": 4.2,
"relevance_score": 0.88
},
{
"action_id": "rank_position_1",
"product_id": "prod_91647",
"product_name": "Samsung Galaxy Buds Pro",
"price": 149.99,
"avg_rating": 4.3,
"relevance_score": 0.85
}
],
"action_probabilities": {
"prod_84729": 0.52,
"prod_52846": 0.28,
"prod_73958": 0.13,
"prod_91647": 0.07
},
"selected_action": {
"product_id": "prod_84729",
"selection_probability": 0.52,
"model_version": "search_ranking_v3.1.8"
},
"outcome": {
"clicked": true,
"click_position": 1,
"time_to_click_seconds": 6.8,
"added_to_cart": false,
"viewed_alternatives": ["prod_52846", "prod_73958"]
}
}
Inspect a structured search log showing context, available actions, product attributes, and probabilities for a candidate set. section 5.5.5
