Human oversight supplies relevance judgments: a person assesses whether a document answers a query or information need. Binary labels commonly mark documents relevant (1) or irrelevant (0), providing training data for learning-to-rank systems. section 10.2 section 10.1.2
Manual judgments establish an initial reference set before automated signals such as clicks are used. Domain understanding matters because no program can generate a reliable golden set without human labeling. section 10.2 section 14.2.3
Movie-search judgments
For the query “star wars,” label Star Wars and Return of the Jedi as relevant, while Star Trek Into Darkness and Battlestar Galactica receive irrelevant labels. Store each judgment with its query and document identifier, then gather feature values for ranking. section 10.2 section 10.1.2
Correcting a silver set
Review model-generated question-answering guesses and classify each row as negative, unanswerable, corrected, correct, or unchecked. The reviewed rows form a higher-quality golden set for later training. section 14.2.3
Human review remains necessary when click-derived judgments contain position or confidence bias. Post-click actions can add evidence, but purchases or other outcomes may reflect factors outside search relevance. section 12.2.4
