Confidence bias occurs when click-derived relevance judgments give observations with little evidence the same influence as observations supported by many examinations. In search ranking, a few lucky clicks can therefore distort learning-to-rank data. section 11.1.4 section 11.3.1
Click models often convert examined and clicked results into grades. A grade based on one examination is statistically fragile, while a grade based on many examinations provides stronger evidence. Training on sparse, spurious grades can therefore alter rankings in the wrong direction. section 11.3.1 section 11.5
A lucky click outranks stronger evidence
For the query “blue ray,” a Blu-ray disc case receives one click from one examination and a perfect grade of 1.000000. A Blu-ray player receives 14 clicks from 34 examinations and a grade of 0.411765. Treating both grades literally ranks the sparsely observed case first, even though the more frequently examined result supplies substantially stronger evidence. section 11.3.1
Reduce confidence bias by incorporating uncertainty into training-data generation. A beta distribution supplies a prior that is updated gradually as new observations arrive, preventing a few clicks from creating an apparently certain judgment. The resulting implicit relevance lists can replace or augment manually created lists in learning-to-rank training. section 11.5
doc_id clicked examined grade name 600603132872 1 1 1.000000 Blu-ray Disc Cases (10-Pack) 827396513927 14 34 0.411765 Panasonic - Blu-Ray Player 25192073007 8 20 0.400000 The Blues Brothers - Widescree... ... 25192107191 0 7 0.000000 Fast Five - Widescreen - Blu-r... 23942972389 0 15 0.000000 Verbatim - 10-Pack 6x BD-R Dis... 885170038875 0 5 0.000000 Panasonic - 9" Widescreen Port...
Inspect the sparse-data judgment table to compare grades with the number of examinations supporting them. section 11.3.1
