A probabilistic judgment represents a search result’s relevance as a probability from 0.0 to 1.0: the estimated chance that a random user considers it relevant. Click models use this form because clicks are noisy, users differ, and search interactions contain bias. section 11.1.1
Movie relevance estimates
For the query “star wars,” Star Wars can receive 0.99, Return of the Jedi 0.80, Star Trek Into Darkness 0.20, and The Star 0.01. These values preserve degrees of relevance instead of forcing every result into relevant or irrelevant. section 11.1.1
Listing 11.2 Labeling movie query relevance probabilistically
# Judgment(grade, keywords, doc_id), sample_judgments = [ Judgment(0.99, "social network", 37799), # The Social Network Judgment(0.01, "social network", 267752), # #chicagoGirl Judgment(0.01, "social network", 38408), # Life As We Know It Judgment(0.01, "social network", 28303), # The Cheyenne Social Club Judgment(0.99, "star wars", 11), # Star Wars Judgment(0.80, "star wars", 1892), # Return of the Jedi Judgment(0.20, "star wars", 54138), # Star Trek Into Darkness Judgment(0.01, "star wars", 85783), # The Star Judgment(0.20, "star wars", 325553) # Battlestar Galactica ]
Inspect a concrete movie-query judgment set with floating-point relevance grades. section 11.1.1
Learning-to-rank systems can use these judgments in several ways: quantize probabilities into discrete labels, pass floating-point grades directly, or preserve only pairwise ordering. For example, 0.80 outranks 0.20, so the model can learn from their feature difference. section 11.1.2
