liveBook · Manning

Recommender system

As explained in AI Model Evaluation

An AI system that selects, predicts, or ranks items for users based on preferences, behavior, or item characteristics.

A recommender system selects and ranks items for a user. In a streaming app, it can reduce browsing time by placing promising movies and shows near the top of a personalized list. section 1.3 section 2.2.3

A recommendation pipeline can create a pool of candidates and rank them using signals such as watch history, inferred preferences, item metadata, and popularity. A genre-based model, for example, infers preferred genres from past interactions and recommends movies with similar characteristics. section 1.3 section 2.2.3

To evaluate a ranking change offline:

  • Collect historical interactions, such as impressions, clicks, watches, saves, and ratings, along with useful user and movie metadata. section 2.2.3
  • Train the candidate model, generate ranked lists for users in a holdout dataset, and compare them with interaction signals used as proxy ground truth. section 2.2.3
  • Choose metrics for the product goal. Precision@K assesses relevance near the top of a list; recall can assess whether relevant past shows are retrieved. NDCG assesses ranking, while diversity and coverage can reveal narrow or repetitive recommendations. section 1.3 section 2.2.3

Precision for a movie list

For one user, treat movies rated at least 4 in the test data as relevant. Count how many appear among the top recommendations and divide by the smaller of K and the number recommended. If there are no relevant test movies, the calculation returns no score for that user. section 2.2.3

Listing 2.1 Compute precision
def calculate_precision_at_k(user_id, test_data, recommendations, k=10):
    """Calculate precision@k for a single user."""
    # Get highly rated movies the user interacted with in the test set
    actual_movies = set(
        test_data[
            (test_data["user_id"] == user_id) &
            (test_data["rating"] >= 4)
        ]["movie_id"]
    )
    
    if not actual_movies:
        return None  # No relevant test data for this user
    
    # Get top-k recommended movies
    recommended_movies = [movie_id for movie_id, _ in recommendations[:k]]
    
    # Calculate precision: number of hits divided by k
    hits = len(set(recommended_movies) & actual_movies)
    precision = hits / min(k, len(recommended_movies)) if recommended_movies else 0
    
    return precision
 

Inspect the per-user Precision@K calculation, including its handling of missing relevant test data and short recommendation lists. section 2.2.3

Historical interactions are imperfect measures of preference: a click may reflect prominent placement rather than genuine relevance. Use offline results to assess changes before deployment, but move to user-facing experimentation when results are promising rather than treating metrics as proof of product quality. section 2.2.3 section 1.3

How AI Model Evaluation covers it

How deep it goes
Explains candidate ranking and a holdout evaluation workflow, including a per-user Precision@K calculation, rather than building a complete recommender.
Examples
Uses streaming homepage recommendations, a genre-based movie model, and a recent-shows retrieval scenario to illustrate different goals and metrics.
What you can do after
Evaluate/understand recommendation ranking with holdout interactions and offline metrics while recognizing that interaction signals imperfectly represent relevance.
How it connects
Connects recommendation ranking with collaborative filtering, offline evaluation, Precision@K, recall, NDCG, diversity, and coverage.

Also explained in

Compare how 11 books explain recommender system →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.