Suppose a search result would satisfy a user, but the ranking never shows it. The user cannot click it, so a learning-to-rank (LTR) model trained on clicks receives no evidence that it is relevant. Presentation bias is this distortion of interaction data by the results the search engine chose to display. section 12.0
The loop can reinforce itself: results with interaction signals rank well, earn more exposure, and collect more signals. Unseen or new documents struggle to gain the signals needed to appear. A better click model alone cannot learn relevance from a result that was never shown. section 4.2.3 section 12.2.4
Look for gaps in logged results
For the query “transformers dvd,” inspect logged judgments alongside features such as name_match, description matches, and has_promotion. Every logged item has a name match, and none has a promotion. That does not prove promoted items or items without a name match are relevant; it shows that the training data cannot answer whether they are. Separately, a simulated A/B test records 21 purchases for one model and 15 for a model that had looked stronger on offline metrics—reason to check live outcomes rather than trust the existing judgments alone. section 12.3.1 section 12.2.3
To address the blind spot:
- Audit logged judgments for missing kinds of results, using features that describe both familiar and unexplored candidates. section 12.3.1
- Show some promising results outside that coverage and use the resulting interactions as new training data. Balance this exploration against exploiting rankings already known to perform well; filling results with random products is not the goal. section 12.0 section 12.3 section 12.3.1
doc_id query grade long_desc*_match name_match
short_desc*_match has_promotion
----------------------------------------------------------------------------
97363560449 transformers dvd 0.34 0.0 0.0 1.0 0.0
97361312804 transformers dvd 0.34 0.0 0.0 1.0 0.0
97361312743 transformers dvd 0.34 0.0 0.0 1.0 0.0
97363455349 transformers dvd 0.34 0.0 0.0 1.0 0.0
...
708056579739 transformers dvd 0.01 1.0 1.0 1.0 0.0
879862003524 transformers dvd 0.01 1.0 1.0 1.0 0.0
93624974918 transformers dvd 0.01 0.0 0.0 1.0 0.0
47875839090 transformers dvd 0.01 1.0 0.0 1.0 0.0
The logged-judgments table makes the coverage gap visible: its rows vary on description matches, but all shown rows have name_match=1.0 and has_promotion=0.0. section 12.3.1
