chapter four

4 Evaluating what makes a good recommendation

 

This chapter covers

  • Evaluation of recommendations
  • Definition of metrics
  • Splitting datasets
  • Offline evaluation

Evaluating recommendations isn't just for competitive data scientists or engineers. The numbers that come out of an evaluation rarely provide an objective answer — they only become meaningful when compared against metrics from other models and systems. This makes evaluation fundamentally a communication tool: a way for engineers and stakeholders to align on whether a change is worth shipping.

Stakeholders don't want to hear about state-of-the-art algorithms from lengthy scientific papers. They want to know whether an improved recommender will attract more loyal users, increase engagement, or boost revenue. These evaluations directly affect strategic decisions — whether to ship a new feature, invest in more AI infrastructure, or stick with what's already working.

Poor evaluation wastes engineering resources on models that don't improve the user experience, or worse, quietly degrade it. Good evaluation helps teams ship effective features confidently and learn quickly from mistakes.

4.1 Why Evaluation is Different for Recommender Systems

In image classification, a picture of a cat is objectively a cat — the ground truth is clear and unchanging. In recommender systems, the "right" recommendation depends on unobservable context that changes from moment to moment and varies across users in ways we can only partially model.

4.1.1 There is no ground Truth

4.1.2 Evaluation is Always Comparative

4.1.3 What Makes a Good Recommendation?

4.2 The Evaluation Process: From Data to Decisions

4.2.1 The Offline Evaluation Workflow

4.2.2 Evaluation as Experimentation

4.2.3 What We're Actually Measuring

4.3 Preparing Data for Evaluation

4.3.1 Temporal Splitting: Simulating the Future

4.3.2 Start with a sample

4.4 Your First Evaluation: A Simple Example

4.4.1 Setting Up the Evaluation

4.4.2 Defining the Relevance Set

4.4.3 Calculating Your First Metric

4.4.4 Adding a Baseline: Popularity

4.5 Establishing Baselines

4.5.1 The Baseline Ladder

4.5.2 Academic vs. Industry Baselines

4.6 Metrics: Three Dimensions of Quality

4.6.1 Accuracy Metrics: Precision, Recall, and Hit Rate

4.6.2 Averages and Distributions

4.6.3 Rank-Aware Metrics

4.6.4 Beyond Accuracy

4.6.5 Evaluating the Full Pipeline

4.7 Users and Their Lifecycle

4.8 Putting It All Together

4.9 Further reading