A relevancy score expresses how well an item matches a user query and provides a basis for ranking search results. In photo search, first filter out images that fail explicit requirements—such as file format, width, or price. Then score the remaining query–image pairs. This avoids ranking millions of photos that the user would reject anyway. section 3.4.2
relevancy = f(encode_text(query), encode_image(item))
The sketch separates text and image encoding from the function that turns their representations into a relevancy score. It leaves the encoding and scoring choices open. section 3.4.2
What should count as a good match? Past queries paired with clicks signal user interest; time on a photo page and purchases can add context. Human labelers can instead assign scores directly to query–image pairs. Randomly selected pairs are a starting point, with less-certain or error-prone pairs candidates for later labeling. Direct labels require a dedicated budget. section 6.7.2
Start with a score that is easy to compare, then decide what extra signals earn their complexity:
- Use query and photo metadata with TF-IDF features and a linear model as a simple ML baseline. It does not use the image itself, but its simplicity helps with early debugging. section 8.6.2
- Try CLIP output for query–image pairs as a visual-text baseline. Query–image and query–description distances can also contribute to a combined score. section 8.6.2
- Leave room for user and document features in a unified score. A simple sum is one starting point; a weighted sum allows a small linear model to learn how much each signal contributes. section 11.5.2
