Retrieval ranking orders candidate items by relevance to a user or query. Offline metrics assess whether useful items appear near the top; relative ranking can also compare several candidate outputs for the same input. section 1.3.4 section 9.5.3
For an offline movie-recommendation evaluation:
- Use historical interactions—such as clicks, watches, ratings, or saves—and relevant user or movie context as inputs. section 2.2.3
- Generate ranked recommendations for users in a holdout set and compare them with interaction signals treated as proxy relevance labels. section 2.2.3
- Measure ranking with metrics such as Precision@K or NDCG; consider diversity and coverage to identify narrow or repetitive recommendations. section 2.2.3
Movie recommendations
If movies rated 4 or higher count as relevant, Precision@K measures the fraction of the top K recommendations that match those movies. Ratings are only a proxy for preference: a prominent placement can prompt a click without being the best recommendation. section 2.2.3
Table 1.1 Offline Metric Categories
|
Metric Category
|
Description
|
Example application
|
Common Metrics
|
|
Classification Metrics
|
To evaluate the performance of models that categorize data into predefined classes.
|
Email spam detection system
|
Accuracy, Precision, Recall, F1 Score
|
|
Ranking and Retrieval Metrics
|
To evaluate models that retrieve, sort, or rank items based on relevance to a user or query.
|
Recommendation systems, search, RAG retrieval
|
NDCG, MAP, Precision@K
|
|
Clustering Metrics
|
To evaluate models that groups data points based on similarities, without predefined labels.
|
Customer segmentation for personalization
|
Silhouette Score, Davies-Bouldin Index
|
|
Agentic System Metrics
|
To evaluate systems that plan, call tools, take actions, or complete multi-step tasks.
|
Support agent assistant, coding agent, workflow automation agent
|
Task completion rate, tool-call accuracy, step success rate, constraint adherence, human override rate
|
|
Natural Language Processing (NLP) Metrics
|
To evaluate models that process, classify, translate, transcribe, summarize, or compare human language.
|
Machine translation, speech-to-text, text classification, summarization
|
BLEU, ROUGE, METEOR, BERTScore
|
|
Generative AI / LLM Metrics
|
To evaluate generated outputs for usefulness, correctness, consistency, and alignment with user intent.
|
Customer support chatbot, summarization assistant, personalized recommendation explanations
|
Relevance, factuality, groundedness, instruction following, toxicity; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
|
|
Time Series and Forecasting Metrics
|
To evaluate models that predict future values or trends based on historical data.
|
Forecasting for sales or inventory management
|
MAE, RMSE, MAPE
|
|
Image and Computer Vision Metrics
|
To evaluate models that analyze visual data, such as images or videos.
|
Product image classification on a website
|
IoU, mAP, Accuracy
|
|
Regression Metrics
|
To evaluate models that predict continuous outcomes.
|
Predicting housing prices
|
RMSE, MAE
|
Compare ranking and retrieval metrics with other offline metric categories and their example uses. section 1.3.4








