liveBook · Manning

Evaluation

As explained in AI Recommender Systems

Evaluation is the systematic assessment of a system or approach by comparing its decisions or outcomes against a reference, alternative, or objective.

Evaluation systematically assesses a system by comparing its decisions or outcomes with a reference, alternative, or objective. In recommendation systems, the reference is not objectively correct: user intent changes with context and remains partly unobservable. Evaluation therefore asks whether one approach makes better decisions than another, building confidence for decisions about shipping, investment, and user experience. section 4.0 section 4.1

A practical recommendation-evaluation workflow includes:

  • Build a relevance set from historical data and split it temporally: train on the past, test on the future, and cover full weeks to preserve behavioral patterns. section 4.10
  • Compare against a baseline ladder: random as a sanity check, popularity as a strong simple baseline, and basic personalization. section 4.10
  • Select metrics for the decision: precision, recall, and hit rate measure relevant items; NDCG, MRR, and MAP measure top-rank placement; coverage, diversity, and serendipity expose beyond-accuracy effects. section 4.10
  • Inspect metric distributions across users, lifecycle stages, and pipeline stages rather than relying only on averages or end-to-end scores. section 4.10

Fast iteration before full evaluation

Sample 50% of users, optionally cap each user's ratings at 10 or 20, and run the complete pipeline. Check recommendations for a known item such as a Star Wars movie, recording expectations such as “if X, then not Y.” Scale to the full dataset only after the implementation and spot checks behave reasonably. section 4.3.2

Listing 4.3 Sampling data
sampled_users = ratings['userId'].sample(frac=0.5, 
                                  random_state=42).unique() #A
 
ratings = ratings[ratings['userId'].isin(sampled_users)]   #B

The listing implements early evaluation sampling by selecting half of the users with a fixed random seed and filtering ratings to those users. section 4.3.2

How AI Recommender Systems covers it

How deep it goes
Develops a practical workflow covering splits, baselines, metrics, sampling, spot checks, pipeline stages, and user lifecycle gaps.
Examples
Uses random, popularity, and personalized baselines, staged metrics, sampled users, capped ratings, and Star Wars recommendation spot checks.
What it assumes
Assumes recommendation data and pipelines, while teaching evaluation uncertainty, relevance sets, baselines, metrics, and practical checks.
What you can do after
Apply it via comparative metrics, temporal splits, baseline ladders, sampled runs, spot checks, and lifecycle-aware analysis.
How it connects
Connects evaluation with recommendation systems, relevance sets, temporal splitting, baselines, ranking metrics, coverage, diversity, and cold-start users.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Recommender Systems. The section links point to the parts of the book it draws on.