liveBook · Manning

Evaluation dataset

As explained in AI Model Evaluation

Data kept separate from training and tuning for a relatively independent assessment of a finalized or mostly finalized model.

An evaluation dataset provides data for assessing a model rather than training it. For a relatively independent final assessment, keep a holdout dataset separate from both training and tuning until the model or approach is mostly finalized. In machine learning, this holdout often corresponds to the test set. section 1.3.3

Validation data serves a different purpose: it helps compare model iterations, select features or hyperparameters, and check for overfitting. Repeated development decisions can indirectly adapt a model to validation data, so it should not also serve as the final assessment dataset. section 1.3.3

Evaluating movie recommendations

Train a genre-based recommender on historical interactions, then generate ranked lists for users in a holdout dataset. Compare the recommendations with clicked, watched, rated, or saved movies as proxy relevance signals. Slice results by user type, including cold-start users and users with sparse histories or niche preferences. A simplified Precision@K calculation treats movies rated at least 4 in the test set as relevant. section 2.2.3

Listing 2.1 Compute precision
def calculate_precision_at_k(user_id, test_data, recommendations, k=10):
    """Calculate precision@k for a single user."""
    # Get highly rated movies the user interacted with in the test set
    actual_movies = set(
        test_data[
            (test_data["user_id"] == user_id) &
            (test_data["rating"] >= 4)
        ]["movie_id"]
    )
    
    if not actual_movies:
        return None  # No relevant test data for this user
    
    # Get top-k recommended movies
    recommended_movies = [movie_id for movie_id, _ in recommendations[:k]]
    
    # Calculate precision: number of hits divided by k
    hits = len(set(recommended_movies) & actual_movies)
    precision = hits / min(k, len(recommended_movies)) if recommended_movies else 0
    
    return precision
 

Inspect a per-user Precision@K calculation using highly rated movies in test data as the relevance signal. section 2.2.3

A separate holdout is not automatically representative. Outdated behavior, exposure bias from a previous model, and incomplete logging can make offline results misleading. Interaction signals are also imperfect: a prominent placement may earn a click without being the best recommendation. Check more than one metric; stronger precision can coexist with narrower or more repetitive recommendations. section 1.3.3 section 2.2.3

For comparisons across teams, establish shared definitions for evaluation datasets, baselines, thresholds, and metrics. Even a shared metric name can hide differences in relevance definitions, K, or filtering logic. Reusable frameworks can support dataset loading, metric computation, logging, versioning, and comparison reports while allowing product-specific choices. section 2.4.1

How AI Model Evaluation covers it

How deep it goes
Defines dataset splits and their purposes, then illustrates holdout-based ranking evaluation and its limitations.
Examples
Uses movie interaction histories and ratings to evaluate ranked recommendations, including a per-user Precision@K calculation.
What you can do after
Evaluate and understand a model using a holdout while recognizing representativeness and metric limitations.
How it connects
Connects holdout data with validation data, backtesting, proxy ground truth, and Precision@K.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.