When someone searches for a movie, finding documents that contain the query words is only the first step. Relevance describes how well each returned document matches that query. A search engine assigns matching documents scores and ranks the strongest matches first; the calculation can be adjusted for a particular query. section 3.0
Keep two uses of relevance distinct:
- A relevance score is produced by a ranking function. Keyword matching, BM25, or cosine similarity between query and document vectors can supply scores for sorting results. A keyword-based score tells you which retrieved documents matched better, not whether any is objectively a good answer. section 3.0 section 13.2.1
- A relevance grade is a label assigned to a query–document pair. A judgment list can use binary grades—0 for irrelevant and 1 for relevant—as training data for learning to rank. Grades can also take other forms. section 10.2
Grade movie results
For the query “star wars,” a judgment list labels Return of the Jedi relevant and Star Trek Into Darkness irrelevant. Each label belongs to that document *for that query*. Features such as title and overview BM25 scores can then be gathered for the labeled documents so a learning-to-rank model can learn a scoring function. section 10.2 section 10.1.1
The candidate set matters as much as its order. In a search request, filters remove documents; the query can both restrict matches and contribute to scoring. If the goal is to change ranking without discarding otherwise eligible results, separate those operations: retain the desired filter and use a boost query to influence relevance. section 3.2.6
User behavior offers another source of grades. A click model takes the displayed query, result rank, document, and click status from search sessions, then estimates how often each result was examined. The simple grade is clicks divided by examinations. Do not treat one click after one examination as equally convincing as many clicks across many examinations: a prior-weighted grade accounts for the amount of evidence. Click behavior also has biases that a click model must address. section 12.1.1 section 11.3
query doc_id clicked examined grade beta_grade
blue ray 27242815414 42 42 1.000000 0.846154
827396513927 1304 3359 0.388211 0.387652
883929140855 140 506 0.276680 0.275194
885170033412 568 2147 0.264555 0.264256
24543672067 665 2763 0.240680 0.240534
...
transformers dvd 47875819733 24 1679 0.014294 0.015394
708056579739 23 1659 0.013864 0.014979
879862003524 23 1685 0.013650 0.014749
93624974918 19 1653 0.011494 0.012628
47875839090 16 1669 0.009587 0.010721
Compare `grade` with `beta_grade` in the query–document training data. The counts show why a relevance estimate needs its supporting evidence, not just a number. section 12.1.1
Relevance is therefore both a ranking decision and a learning target. Manually tuned boosts can combine search features, while learning to rank learns their weights from judgments. Click-derived grades can scale that training, but their confidence and bias need attention before they are trusted. section 10.1.1 section 12.1.1
