Evaluation systematically assesses a system by comparing its decisions or outcomes with a reference, alternative, or objective. In recommendation systems, the reference is not objectively correct: user intent changes with context and remains partly unobservable. Evaluation therefore asks whether one approach makes better decisions than another, building confidence for decisions about shipping, investment, and user experience. section 4.0 section 4.1
A practical recommendation-evaluation workflow includes:
- Build a relevance set from historical data and split it temporally: train on the past, test on the future, and cover full weeks to preserve behavioral patterns. section 4.10
- Compare against a baseline ladder: random as a sanity check, popularity as a strong simple baseline, and basic personalization. section 4.10
- Select metrics for the decision: precision, recall, and hit rate measure relevant items; NDCG, MRR, and MAP measure top-rank placement; coverage, diversity, and serendipity expose beyond-accuracy effects. section 4.10
- Inspect metric distributions across users, lifecycle stages, and pipeline stages rather than relying only on averages or end-to-end scores. section 4.10
Fast iteration before full evaluation
Sample 50% of users, optionally cap each user's ratings at 10 or 20, and run the complete pipeline. Check recommendations for a known item such as a Star Wars movie, recording expectations such as “if X, then not Y.” Scale to the full dataset only after the implementation and spot checks behave reasonably. section 4.3.2
Listing 4.3 Sampling data
sampled_users = ratings['userId'].sample(frac=0.5,
random_state=42).unique() #A
ratings = ratings[ratings['userId'].isin(sampled_users)] #BThe listing implements early evaluation sampling by selecting half of the users with a fixed random seed and filtering ratings to those users. section 4.3.2




























