A two-stage recommendation architecture first retrieves a manageable set of candidate items, then ranks those candidates using additional signals. This separates finding potentially relevant items from deciding which ones to show and in what order. section 7.2
Ranking movies for a streaming row
A two-tower candidate model retrieves a few hundred movies using user embedding similarity and metadata. A gradient-boosted decision tree then combines personalization, popularity, recency, freshness, and business rules to rank approximately 20 titles for display. section 7.2
Before an A/B test, an internal beta can compare the new model’s lists with the production system. Employees rate relevance, diversity, and novelty and flag odd picks. This can expose problems that confident offline metrics alone may not reveal. section 7.2
Figure 7.1 illustrates an example of a systemic ranking bias caught during internal beta testing: The same popular title (represented as a star) repeatedly appears in the top position of the 'Movies To Watch' row across different users and segments. This pattern, observed before launch, indicates an overweighted popularity feature in the re-ranking stage which is a problem that could influence engagement metrics if not caught before.
A repeatedly top-ranked popular title across users can reveal an overweighted popularity feature in the re-ranking stage. section 7.2
