A recommender system selects and ranks items for a user. In a streaming app, it can reduce browsing time by placing promising movies and shows near the top of a personalized list. section 1.3 section 2.2.3
A recommendation pipeline can create a pool of candidates and rank them using signals such as watch history, inferred preferences, item metadata, and popularity. A genre-based model, for example, infers preferred genres from past interactions and recommends movies with similar characteristics. section 1.3 section 2.2.3
To evaluate a ranking change offline:
- Collect historical interactions, such as impressions, clicks, watches, saves, and ratings, along with useful user and movie metadata. section 2.2.3
- Train the candidate model, generate ranked lists for users in a holdout dataset, and compare them with interaction signals used as proxy ground truth. section 2.2.3
- Choose metrics for the product goal. Precision@K assesses relevance near the top of a list; recall can assess whether relevant past shows are retrieved. NDCG assesses ranking, while diversity and coverage can reveal narrow or repetitive recommendations. section 1.3 section 2.2.3
Precision for a movie list
For one user, treat movies rated at least 4 in the test data as relevant. Count how many appear among the top recommendations and divide by the smaller of K and the number recommended. If there are no relevant test movies, the calculation returns no score for that user. section 2.2.3
Listing 2.1 Compute precision
def calculate_precision_at_k(user_id, test_data, recommendations, k=10):
"""Calculate precision@k for a single user."""
# Get highly rated movies the user interacted with in the test set
actual_movies = set(
test_data[
(test_data["user_id"] == user_id) &
(test_data["rating"] >= 4)
]["movie_id"]
)
if not actual_movies:
return None # No relevant test data for this user
# Get top-k recommended movies
recommended_movies = [movie_id for movie_id, _ in recommendations[:k]]
# Calculate precision: number of hits divided by k
hits = len(set(recommended_movies) & actual_movies)
precision = hits / min(k, len(recommended_movies)) if recommended_movies else 0
return precision
Inspect the per-user Precision@K calculation, including its handling of missing relevant test data and short recommendation lists. section 2.2.3
Historical interactions are imperfect measures of preference: a click may reflect prominent placement rather than genuine relevance. Use offline results to assess changes before deployment, but move to user-facing experimentation when results are promising rather than treating metrics as proof of product quality. section 2.2.3 section 1.3










