A scoring mechanism assigns numerical values to one or more dimensions of a model output. It turns qualities such as factuality, helpfulness, safety, or task success into signals that can be aggregated and compared. The useful score is the one aligned with the product decision: for example, whether a recommendation is relevant or whether an agent completes its task reliably. section 1.3.4
LLM-as-a-judge is a scoring method, not a metric category. A judge can score groundedness, relevance, factual consistency, instruction following, or task success using a rubric or grading prompt. Treat its output as an evaluation signal rather than absolute truth: validate and calibrate it before using it to guide model or product decisions. section 1.3.4
Choose the output format according to the decision you need:
- Use scoring, such as 1–5 values, to track aggregate trends across many examples, including average factuality, helpfulness, or safety. section 9.5.3
- Use ranking to order several candidates for the same input, such as recommendations, summaries, or agent plans; account for presentation order, length, and style effects. section 9.5.3
- Use pairwise comparison to decide whether one response, model variant, or prompt beats another; expect higher judge-call costs at scale. section 9.5.3
Recommendation evaluation
For a recommendation system, score each output for relevance and aggregate the results across users to monitor trends. Compare two model variants pairwise when choosing a new baseline, then inspect individual failures rather than treating the average as complete evidence of user value. section 1.3.4 section 9.5.3
A number can hide the reason for failure. A response with one dangerous hallucination may receive a similar score to a response that is broadly good but error-free. Combine checklist items for concrete requirements such as factuality or policy compliance with scores for holistic qualities such as tone or helpfulness. section 10.2.1
Table 1.1 Offline Metric Categories
|
Metric Category
|
Description
|
Example application
|
Common Metrics
|
|
Classification Metrics
|
To evaluate the performance of models that categorize data into predefined classes.
|
Email spam detection system
|
Accuracy, Precision, Recall, F1 Score
|
|
Ranking and Retrieval Metrics
|
To evaluate models that retrieve, sort, or rank items based on relevance to a user or query.
|
Recommendation systems, search, RAG retrieval
|
NDCG, MAP, Precision@K
|
|
Clustering Metrics
|
To evaluate models that groups data points based on similarities, without predefined labels.
|
Customer segmentation for personalization
|
Silhouette Score, Davies-Bouldin Index
|
|
Agentic System Metrics
|
To evaluate systems that plan, call tools, take actions, or complete multi-step tasks.
|
Support agent assistant, coding agent, workflow automation agent
|
Task completion rate, tool-call accuracy, step success rate, constraint adherence, human override rate
|
|
Natural Language Processing (NLP) Metrics
|
To evaluate models that process, classify, translate, transcribe, summarize, or compare human language.
|
Machine translation, speech-to-text, text classification, summarization
|
BLEU, ROUGE, METEOR, BERTScore
|
|
Generative AI / LLM Metrics
|
To evaluate generated outputs for usefulness, correctness, consistency, and alignment with user intent.
|
Customer support chatbot, summarization assistant, personalized recommendation explanations
|
Relevance, factuality, groundedness, instruction following, toxicity; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
|
|
Time Series and Forecasting Metrics
|
To evaluate models that predict future values or trends based on historical data.
|
Forecasting for sales or inventory management
|
MAE, RMSE, MAPE
|
|
Image and Computer Vision Metrics
|
To evaluate models that analyze visual data, such as images or videos.
|
Product image classification on a website
|
IoU, mAP, Accuracy
|
|
Regression Metrics
|
To evaluate models that predict continuous outcomes.
|
Predicting housing prices
|
RMSE, MAE
|
Use the metric-category table to match the scoring target to the system being evaluated, from ranking and retrieval to agentic task completion. section 1.3.4
