liveBook · Manning

Information retrieval

As explained in AI Recommender Systems

A candidate-generation technique that finds potentially relevant items or documents from a larger collection.

Information retrieval finds a manageable pool of potentially relevant items in a much larger collection. In a recommender system, retrieval is the fast first stage: it might select hundreds or thousands of candidates from millions of items. It aims to include good possibilities, not to produce the final ranked list. section 1.4.1 section 4.6.5

A four-stage pipeline gives retrieval a distinct job:

  • Retrieve candidates quickly, using a method such as a database query or approximate nearest-neighbor search. section 1.4.1
  • Filter out ineligible items, such as previously consumed or unavailable products. section 1.4.1
  • Score the remaining candidates for relevance, then order a final list using scores and presentation rules. section 1.4.1

Popular content, personalized later

A streaming carousel can begin with a query for titles that drew many views recently. That query might return 500–1,000 popular titles shared by all users. Filtering, scoring, and ordering then turn this broad pool into different recommendations for different users. An e-commerce page can instead retrieve products frequently bought with the product being viewed. section 1.4.1 section 1.4.2

Listing 2.14 Four-stage recommender
def recommend_for_user(user_id, k=10):
  candidates = retrieve_candidates_for_user(user_id) #A
  candidates = apply_filters(candidates, user_id) #B
  candidates = add_additional_scores(candidates) #C
  recommendations = rank_and_select_top_k(candidates, k) #D
  return recommendations
 

The recommender listing makes the boundary concrete: retrieve_candidates_for_user supplies candidates before filtering, additional scoring, and top-k selection. section 2.2.8

Evaluate retrieval by recall: did relevant items enter the candidate pool? Their positions within that pool matter less than their presence because later stages reorder them. Measure recall at several pool sizes, such as 50, 100, 200, and 500, to find a useful cap before larger pools bring diminishing returns. A final-list metric alone cannot tell whether a decline came from retrieval, scoring, or filtering. section 4.6.5

Retrieval methods can be swapped without replacing the rest of the pipeline. ItemKNN, content-based signals, and popularity provide different starting points; a popularity fallback can supply candidates when a user has no history or a model fails. Optimize retrieval with downstream results in mind: a fast stage is not useful if it omits items the scorer could have ranked well. section 2.2.8 section 2.6

How AI Recommender Systems covers it

How deep it goes
Explains pipeline placement, interchangeable implementations, and recall-based evaluation; defers construction of the pool-size measurement.
Examples
Uses a popular-content carousel, co-purchased products, and a four-stage recommender function to illustrate candidate generation.
What it assumes
Builds the four-stage structure from a simple recommender while using user ratings and item similarity in its implementation example.
What you can do after
Build it from scratch as a pipeline stage, swap retrieval methods, and diagnose missed candidates with recall at multiple pool sizes.
How it connects
Connects candidate generation to collaborative filtering, content-based recommendations, filtering, scoring, ranking, and retrieval-augmented generation.

Also explained in

Compare how 17 books explain information retrieval →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.