When a search engine returns photos, a practical question is how quickly users encounter a useful result. Mean reciprocal rank (MRR) measures this across queries: for each query, take the reciprocal of the rank of its first relevant result, then average those values. It rewards putting the first useful result near the top. section 5.3.2
What the score rewards
If the first relevant photo is ranked first for one query and second for another, their reciprocal ranks are 1 and 1/2; MRR across the two queries is 0.75. The calculation focuses on the first relevant result, not every useful photo further down the list. section 5.3.2
MRR is a fast offline metric, but it needs a relevance signal for each query. Human-labeled relevant results can support an MRR intended as a proxy for click-through rate (CTR). That does not automatically make it a proxy for purchase conversion: crowdsourced relevance labels may not reflect what customers buy. Purchase-based evaluation needs real purchase data, while conversion may initially have to be monitored online through A/B tests and gradual rollout. section 5.3.2
Choose what counts as the first relevant result before comparing systems:
- MRR (human) uses the first photo labeled relevant. section 12.4.2
- MRR (click) uses the first photo clicked on the search results page. section 12.4.2
- MRR (purchase) uses the first photo purchased from the search results page. section 12.4.2
An offline score is most useful when it predicts a live outcome. One evaluation plan applies candidate metrics to results from an existing non-ML search algorithm, measures CTR and conversion rate over time, and uses Spearman’s rank correlation to select offline metrics most closely associated with each online measure. That check matters before treating an MRR improvement as evidence of business impact. section 12.4.2
