A performance evaluation measures how well a model or AI system achieves its intended objective using predefined metrics. It helps compare model versions and asks: Is the system doing the job it was designed to do? section 2.1.1
Choose metrics that reflect the product goal. A strong benchmark result alone does not establish how a system will behave with real-world data, edge cases, or production constraints. Offline evaluation estimates impact before exposing a change to users; online evaluation tests it through real user interactions. section 1.1 section 1.3.4
Match the performance measure to the system’s job:
- For recommendation and search, use ranking and retrieval metrics such as Precision@K, Recall@K, or NDCG to assess relevant results. section 2.1.1
- For classification, consider accuracy, precision, recall, or F1 score. section 2.1.1
- For LLM features, assess qualities such as relevance, groundedness, factuality, or instruction following. For agents, consider task completion and tool-call accuracy. section 2.1.1
Table 2.1 Common Types of Performance Evaluations
|
Type of Performance Evaluation
|
Purpose
|
Example
|
|
Classification Performance
|
Measures how well a model assigns items to the correct predefined class.
|
Accuracy, Precision, Recall, F1 Score, AUC
|
|
Ranking and Retrieval Performance
|
Measures how well a system retrieves or ranks relevant items, documents, or recommendations.
|
Precision@K, Recall@K, NDCG, MAP, MRR
|
|
Generative AI / LLM Output Performance
|
Measures whether generated outputs satisfy the intended task or user need.
|
Relevance, groundedness, factuality, instruction following, answer completeness; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
|
|
Agentic Task Performance
|
Measures whether an agent successfully completes a task, calls the right tools, and follows constraints.
|
Task completion rate, tool-call accuracy, step success rate, constraint adherence
|
|
Engineering System Performance
|
Measures whether the model or AI system can run within production constraints.
|
Inference latency, throughput, resource utilization, token cost, tool-call latency, error rate
|
Compare performance evaluation types, their purposes, and example metrics. section 2.1.1
Movie recommendations
For each user, compare the recommended movies with the movies the user would watch. Recall is the number in both sets divided by the total the user would watch. Average recall summarizes retrieval performance, but it can hide much lower recall for users who prefer foreign-language films. Slice the results to find which users are underserved. section 3.1.2
Listing 3.2 Computing the Offline Evaluation Recall Metric
# 7.2 Calculate recall for each user
# Recall = (number of recommended movies user would watch) / (total number of movies user would watch)
user_recall = []
for user_id, preferred_movies in user_preferences.items():
# Get recommendations for this user
recommended_movies = recommendations_df[recommendations_df['user_id'] == user_id]['movie_id'].tolist()
# Calculate intersection (recommended AND would watch)
relevant_recommendations = set(recommended_movies).intersection(set(preferred_movies))
# Calculate recall
recall = len(relevant_recommendations) / len(preferred_movies) if preferred_movies else 0
# Get user demographics
user_info = users_df[users_df['user_id'] == user_id].iloc[0]
user_recall.append({
'user_id': user_id,
'age_group': user_info['age_group'],
'subscription_type': user_info['subscription_type'],
'recall': recall
})
# Convert to DataFrame
recall_df = pd.DataFrame(user_recall)
# Display overall recall
overall_recall = recall_df['recall'].mean()
print(f"Overall Recall: {overall_recall:.4f}")
recall_df.head()
Inspect a per-user recall calculation that also records demographic fields for diagnostic analysis. section 3.1.2
Performance metrics say whether a model meets an objective, but usually not why it succeeds or fails. Diagnostic evaluations break results down by users, items, contexts, or behaviors; simulation and replay can explore alternative conditions. Combining these views gives a more complete picture than an aggregate score alone. section 2.1.1 section 2.1 section 3.1.2
Keep the metric separate from the scoring method. Groundedness, for example, is a dimension to measure; an LLM judge is one way to score it. Validate and calibrate judge outputs, and review disagreements with human labels rather than treating a fluent explanation as proof. section 1.3.4 section 9.7.6
Evaluate deployability alongside model quality. A model that takes 1,500 ms to respond is not viable under a 400 ms latency SLA, however accurate it is. Consider frontend, backend, inference, and infrastructure measures together, including trade-offs among quality, latency, reliability, and cost. section 2.1.1 section 4.2.4
Make comparisons reproducible: version datasets, model artifacts, metric definitions, evaluation code, and configurations. Use traceable runs and reports, and agree on metric definitions across teams—two Precision@K results are not necessarily comparable if relevance, K, or filtering differs. section 2.4 section 2.4.1




























