liveBook · Manning

Precision@k

As explained in AI Model Evaluation

A ranking metric measuring the proportion of relevant items among the top K retrieved or recommended items.

Precision@k measures the proportion of relevant items among the first k recommendations or retrieved results. Use it when the quality of the top results matters most: three relevant titles in five recommendations produce P@5 = 3/5, or 60%. section 1.3.5

Apply Precision@k within an offline evaluation rather than treating it as a complete product-quality signal:

  • Generate ranked recommendation lists for users in a holdout dataset. section 2.2.3
  • Define relevance with an available proxy, such as a high rating, watch, click, save, or meaningful engagement action. section 2.2.3
  • Count relevant items in the top k and divide by the evaluated recommendation count. section 2.2.3
  • Slice results across cold-start users, sparse histories, and niche preferences to expose uneven performance. section 2.2.3
  • Pair the metric with diversity, coverage, novelty, satisfaction, latency, or NDCG when repetition, concentration, position, or system constraints matter. section 1.3.5 section 2.2.3

Movie recommendations

Suppose five TV shows are recommended and three match the user's prior watch history or labeled preferences. Precision@5 is 60%. A test-set implementation can define relevance as a rating of 4 or higher, extract the user's relevant movies, compare them with the top-k recommendations, and return the hit count divided by the evaluated list length. Users without relevant test data should be excluded or handled explicitly.

How AI Model Evaluation covers it

How deep it goes
Defines and computes the metric through a movie example, but stops short of treating relevance or offline scores as complete product evaluation.
Examples
Uses five recommended shows with three relevant items and a Python movie-rating evaluation using a threshold of 4 or higher.
What it assumes
Assumes ranked recommendations, holdout interaction data, and a chosen relevance proxy such as ratings, watches, clicks, or saves.
What you can do after
Apply it via an offline ranking pipeline to compare recommendation changes and identify top-result relevance across user segments.
How it connects
Connects ranking quality with Recall@K, NDCG@K, MAP, diversity, coverage, novelty, satisfaction, latency, and system performance.

Also explained in

Compare how 3 books explain Precision@k →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.