Position bias is the tendency for users to click higher-ranked search results more often than lower-ranked results, even when lower results are more relevant. Trust in the search engine, top-to-bottom scanning, and limited visibility all contribute. section 11.2.1
Raw click-through rate (CTR) therefore mixes relevance with rank. A learning-to-rank model trained directly on clicks can reinforce the existing ordering instead of learning which documents best satisfy the query. section 11.2 section 11.2.2
A practical workflow for detecting and reducing position bias:
- Measure CTR by rank across sessions; sharply higher CTR near the top indicates position bias. section 11.2.2
- Compare average document rank with click-derived judgments to identify results rewarded mainly because they are displayed prominently. section 11.2.2
- Model an examine event: estimate whether a user consciously considered a result before interpreting its click or non-click. section 11.2.3
- Aggregate clicks and examined results by document, then calculate clicks divided by examines as a less position-sensitive grade. section 11.2.3
- Use the resulting implicit judgments in learning-to-rank training, while checking for remaining confidence, presentation, attractiveness, and performance biases. section 11.1.4 section 11.5
Rank-dependent clicks
Across search sessions, CTR falls from 0.249727 at rank 0 to 0.142673 at rank 1, 0.084218 at rank 2, and 0.017364 at rank 9. For “transformers dark of the moon,” highly ranked results such as The A-Team and Fast Five attract clicks despite being irrelevant, misleading a CTR-based model.
Listing 11.7 Marking which results were examined in each session
def calculate_examine_probability(sessions):
last_click_per_session = sessions.groupby( #1
["clicked", "sess_id"])["rank"].max()[True] #1
sessions["last_click_rank"] = last_click_per_session #2
sessions["examined"] = \ #3
sessions["rank"] <= sessions["last_click_rank"] #3
return sessions
sessions = get_sessions("dryer")
probability_data = calculate_examine_probability(sessions).loc[3]
print(probability_data)
This listing demonstrates a simple session-level examine rule: mark results at or above the last clicked rank as examined, then use those observations to adjust click judgments. section 11.2.3
