liveBook · Manning

Two-stage architecture

As explained in AI Model Evaluation

A recommendation architecture that first generates candidate items and then ranks a smaller set using additional signals.

A two-stage recommendation architecture first retrieves a manageable set of candidate items, then ranks those candidates using additional signals. This separates finding potentially relevant items from deciding which ones to show and in what order. section 7.2

Ranking movies for a streaming row

A two-tower candidate model retrieves a few hundred movies using user embedding similarity and metadata. A gradient-boosted decision tree then combines personalization, popularity, recency, freshness, and business rules to rank approximately 20 titles for display. section 7.2

Before an A/B test, an internal beta can compare the new model’s lists with the production system. Employees rate relevance, diversity, and novelty and flag odd picks. This can expose problems that confident offline metrics alone may not reveal. section 7.2

Figure 7.1 illustrates an example of a systemic ranking bias caught during internal beta testing: The same popular title (represented as a star) repeatedly appears in the top position of the 'Movies To Watch' row across different users and segments. This pattern, observed before launch, indicates an overweighted popularity feature in the re-ranking stage which is a problem that could influence engagement metrics if not caught before.

A repeatedly top-ranked popular title across users can reveal an overweighted popularity feature in the re-ranking stage. section 7.2

How AI Model Evaluation covers it

How deep it goes
Explains the retrieval and re-ranking stages through model choices and signals, without implementation code.
Examples
Uses a streaming movie row with hundreds of candidates narrowed to approximately 20 displayed titles.
What you can do after
Evaluate the architecture’s recommendations using internal comparisons and recognize a possible popularity-weighting problem.
How it connects
Connects candidate generation, two-tower models, gradient-boosted decision trees, and ranking bias.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.