liveBook · Manning

Top-k evaluation

As explained in AI Model Evaluation

Top-k evaluation assesses only the first K results in a ranked output list.

Top-k evaluation measures a ranked output by inspecting only its first K results. It focuses evaluation on the recommendations users are most likely to see and interact with, rather than treating every ranked position equally. section 1.3.5

Choose the metric according to what matters in the visible recommendation set:

  • Precision@K asks how many of the top K recommended items are relevant. section 1.3.5
  • Recall@K asks how many relevant items appear within the top K, helping identify missed options. section 1.3.5
  • NDCG@K evaluates whether highly relevant items appear higher in the ranked list, making position important. section 1.3.5
  • Select K using product context, interpretability, and the amount of the list users realistically inspect; evaluating position 78 may add little value. section 1.3.5

Five-show recommendation list

A system recommends five TV shows, three of which are relevant according to watch history or labeled preferences. Precision@5 is 3/5, or 60%. section 1.3.5

Table 1.2 Example of Offline Metrics for a Recommender Model
Metric
Metric Category
Definition
Precision at K (P@K)
Ranking and Retrieval Metric
Of the top K recommended items, how many are relevant? Useful when the quality of the top results matters most.
Recall at K (R@K)
Ranking and Retrieval Metric
Of all relevant items, how many appear in the top K? Useful when the system should avoid missing relevant options.
NDCG@K
Ranking Metric
Whether highly relevant items appear higher in the ranked list. Useful when position matters.
Coverage
Recommender System Metric
How much of the catalog or user base the recommender meaningfully serves. Useful for detecting over-concentration.
Diversity
Recommender System Metric
Whether the recommendation set contains varied items rather than many similar ones. Useful for avoiding repetitive experiences.
Groundedness
Generative AI / LLM Metric
Whether a generated recommendation explanation is supported by user history, item metadata, or retrieved context.

Compare top-k ranking measures with coverage, diversity, and other offline recommender metrics before choosing an evaluation target. section 1.3.5

How AI Model Evaluation covers it

How deep it goes
Defines top-k evaluation and compares Precision@K, Recall@K, and NDCG@K without developing their computation beyond a simple example.
Examples
Uses a five-show recommendation list with three relevant items to calculate Precision@5 as 60%.
What it assumes
Assumes ranked recommendations and relevance labels or prior watch history, while defining the meaning of K from the metric notation.
What you can do after
Evaluate a ranked recommendation list with Precision@K, Recall@K, or NDCG@K according to product priorities.
How it connects
Connects top-k evaluation to precision, recall, NDCG, ranking metrics, recommender systems, and offline evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.