liveBook · Manning

Relevance

As explained in AI Model Evaluation

The degree to which retrieved, recommended, or generated results address the intended query, context, task, or user need.

Relevance is how well a result addresses a user’s query, interests, or task. In a ranked movie recommendation list, more relevant movies should ideally appear near the top so users can find something they want to watch quickly. For a generated answer, relevance is a separate evaluation dimension from groundedness. section 2.2.3 section 1.3.4

Choose a measure that matches the product question. Ranking and retrieval evaluations can use Precision@K or NDCG, while generated answers can be assessed with a relevance rubric. An LLM-as-a-judge is a scoring method, not a metric itself; its relevance judgments are evaluation signals that need validation and calibration. section 1.3.4 section 2.2.3

Offline evaluation of movie recommendations

Train a genre-based model on historical interactions, generate ranked lists for users in a holdout dataset, and compare recommendations with movies they watched, clicked, rated, or saved. These interactions serve as proxy relevance labels, not perfect preferences: a user may click a movie because it was prominently placed. Add diversity or coverage metrics to check whether relevant-looking lists are too narrow or repetitive. section 2.2.3

Listing 2.1 Compute precision
def calculate_precision_at_k(user_id, test_data, recommendations, k=10):
    """Calculate precision@k for a single user."""
    # Get highly rated movies the user interacted with in the test set
    actual_movies = set(
        test_data[
            (test_data["user_id"] == user_id) &
            (test_data["rating"] >= 4)
        ]["movie_id"]
    )
    
    if not actual_movies:
        return None  # No relevant test data for this user
    
    # Get top-k recommended movies
    recommended_movies = [movie_id for movie_id, _ in recommendations[:k]]
    
    # Calculate precision: number of hits divided by k
    hits = len(set(recommended_movies) & actual_movies)
    precision = hits / min(k, len(recommended_movies)) if recommended_movies else 0
    
    return precision
 

This implementation calculates a user’s Precision@K using movies rated at least 4 in the test data as relevant items. It returns no score when the user has no relevant test data. section 2.2.3

A structured LLM judge can rate a user’s top five recommendations for relevance, diversity, and discovery separately. Give it the viewing history, define each criterion, request a short justification and a 1–5 score, and require JSON output. Check stability across reruns, agreement between models, or correlation with human ratings; gold-standard examples can help calibrate judgments. section 9.6.1

Use a fixed dataset and consistent metric definitions to compare model versions, then inspect user and item segments for problems the aggregate score may hide. An offline relevance improvement alone does not establish a better live product experience; diagnostic evaluations and A/B testing address different parts of that question. section 1.4

How AI Model Evaluation covers it

How deep it goes
Works through offline recommendation evaluation and a structured judge rubric, while cautioning that benchmarks may miss live product behavior.
Examples
Uses a genre-based movie recommender, interaction-based proxy labels, Precision@K, and a JSON-scoring judge prompt.
What you can do after
Evaluate recommendation relevance with offline ranking metrics and structured judge rubrics while recognizing proxy-label and benchmark limitations.
How it connects
Connects relevance to ranking, retrieval, groundedness, diversity, and LLM-as-a-judge evaluation.

Also explained in

Compare how 17 books explain relevance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.