A judgment list is a dataset of query-document pairs with relevance labels. A binary list uses 0 for irrelevant and 1 for relevant, while other systems can use graded relevance. Each judgment can also carry a feature vector for learning to rank. section 10.2
Build a judgment list by connecting relevance decisions to search features:
- Group labeled documents by query, such as one relevant and three irrelevant movies for “social network.” section 10.2
- Request feature values for every judged document, passing the query and document IDs to the search engine. section 10.3.2
- Store the returned values in the feature vector, preserving the feature-store order, such as title BM25, overview BM25, and release year. section 10.3.2
- Scale the same process from small manually labeled samples to larger query sets used for model training. section 10.3.2
Listing 10.3 Labeling movie judgments as relevant or irrelevant
sample_judgments = [ # for 'social network' query Judgment(1, "social network", 37799), # The Social Network Judgment(0, "social network", 267752), # #chicagoGirl Judgment(0, "social network", 38408), # Life As We Know It Judgment(0, "social network", 28303), # The Cheyenne Social Club # for 'star wars' query Judgment(1, "star wars", 11), # Star Wars Judgment(1, "star wars", 1892), # Return of the Jedi Judgment(0, "star wars", 54138), # Star Trek Into Darkness Judgment(0, "star wars", 85783), # The Star Judgment(0, "star wars", 325553) # Battlestar Galactica ]
Inspect a compact Python dataset containing binary movie judgments for the queries “social network” and “star wars.” section 10.2
From clicks to graded judgments
Manual labels are one starting point. A click model can derive training labels from sessions containing the query, displayed rank, document, and click status. A simple CTR model groups sessions by document, divides clicks by sessions, and ranks documents by the resulting rate. More advanced processing can produce grade and beta_grade probabilities while accounting for examined results and evidence strength. section 11.1.3 section 12.1.1
Listing 10.6 Judgments with logged features for query social network
[Judgment(grade=1, keywords="social network", doc_id=37799, qid=1,
features=[8.243603, 3.8143613, 2010.0], weight=1), #1
Judgment(0, "social network", 267752, 1, [0.0, 6.0172443, 2013.0], 1),
Judgment(0, "social network", 38408, 1, [0.0, 4.353118, 2010.0],1), #2
Judgment(0, "social network", 28303, 1, [3.4286604, 3.1086721, 1970.0], 1)]
Compare judgments before and after feature logging for “social network,” including grades, query IDs, feature vectors, and weights. section 10.3.2
