liveBook · Manning

Evaluation

As explained in AI Model Evaluation

A performance evaluation measures how well a model or AI system achieves its intended objective against predefined metrics.

A performance evaluation measures how well a model or AI system achieves its intended objective using predefined metrics. It helps compare model versions and asks: Is the system doing the job it was designed to do? section 2.1.1

Choose metrics that reflect the product goal. A strong benchmark result alone does not establish how a system will behave with real-world data, edge cases, or production constraints. Offline evaluation estimates impact before exposing a change to users; online evaluation tests it through real user interactions. section 1.1 section 1.3.4

Match the performance measure to the system’s job:

  • For recommendation and search, use ranking and retrieval metrics such as Precision@K, Recall@K, or NDCG to assess relevant results. section 2.1.1
  • For classification, consider accuracy, precision, recall, or F1 score. section 2.1.1
  • For LLM features, assess qualities such as relevance, groundedness, factuality, or instruction following. For agents, consider task completion and tool-call accuracy. section 2.1.1
Table 2.1 Common Types of Performance Evaluations
Type of Performance Evaluation
Purpose
Example
Classification Performance
Measures how well a model assigns items to the correct predefined class.
Accuracy, Precision, Recall, F1 Score, AUC
Ranking and Retrieval Performance
Measures how well a system retrieves or ranks relevant items, documents, or recommendations.
Precision@K, Recall@K, NDCG, MAP, MRR
Generative AI / LLM Output Performance
Measures whether generated outputs satisfy the intended task or user need.
Relevance, groundedness, factuality, instruction following, answer completeness; often scored through human review, rubric-based evaluation, or LLM-as-a-judge
Agentic Task Performance
Measures whether an agent successfully completes a task, calls the right tools, and follows constraints.
Task completion rate, tool-call accuracy, step success rate, constraint adherence
Engineering System Performance
Measures whether the model or AI system can run within production constraints.
Inference latency, throughput, resource utilization, token cost, tool-call latency, error rate

Compare performance evaluation types, their purposes, and example metrics. section 2.1.1

Movie recommendations

For each user, compare the recommended movies with the movies the user would watch. Recall is the number in both sets divided by the total the user would watch. Average recall summarizes retrieval performance, but it can hide much lower recall for users who prefer foreign-language films. Slice the results to find which users are underserved. section 3.1.2

Listing 3.2 Computing the Offline Evaluation Recall Metric
# 7.2 Calculate recall for each user
# Recall = (number of recommended movies user would watch) / (total number of movies user would watch)
 
user_recall = []
for user_id, preferred_movies in user_preferences.items():
    # Get recommendations for this user
    recommended_movies = recommendations_df[recommendations_df['user_id'] == user_id]['movie_id'].tolist()
    
    # Calculate intersection (recommended AND would watch)
    relevant_recommendations = set(recommended_movies).intersection(set(preferred_movies))
    
    # Calculate recall
    recall = len(relevant_recommendations) / len(preferred_movies) if preferred_movies else 0
    
    # Get user demographics
    user_info = users_df[users_df['user_id'] == user_id].iloc[0]
    
    user_recall.append({
        'user_id': user_id,
        'age_group': user_info['age_group'],
        'subscription_type': user_info['subscription_type'],
        'recall': recall
    })
 
# Convert to DataFrame
recall_df = pd.DataFrame(user_recall)
 
# Display overall recall
overall_recall = recall_df['recall'].mean()
print(f"Overall Recall: {overall_recall:.4f}")
recall_df.head()
 

Inspect a per-user recall calculation that also records demographic fields for diagnostic analysis. section 3.1.2

Performance metrics say whether a model meets an objective, but usually not why it succeeds or fails. Diagnostic evaluations break results down by users, items, contexts, or behaviors; simulation and replay can explore alternative conditions. Combining these views gives a more complete picture than an aggregate score alone. section 2.1.1 section 2.1 section 3.1.2

Keep the metric separate from the scoring method. Groundedness, for example, is a dimension to measure; an LLM judge is one way to score it. Validate and calibrate judge outputs, and review disagreements with human labels rather than treating a fluent explanation as proof. section 1.3.4 section 9.7.6

Evaluate deployability alongside model quality. A model that takes 1,500 ms to respond is not viable under a 400 ms latency SLA, however accurate it is. Consider frontend, backend, inference, and infrastructure measures together, including trade-offs among quality, latency, reliability, and cost. section 2.1.1 section 4.2.4

Make comparisons reproducible: version datasets, model artifacts, metric definitions, evaluation code, and configurations. Use traceable runs and reports, and agree on metric definitions across teams—two Precision@K results are not necessarily comparable if relevance, K, or filtering differs. section 2.4 section 2.4.1

How AI Model Evaluation covers it

How deep it goes
Defines performance evaluation, compares complementary methods, and gives metric, diagnostic, and infrastructure guidance without treating scores as explanations.
Examples
Calculates movie recommendation recall and examines how aggregate results can conceal differences between user groups.
What you can do after
Evaluate model versions against product-aligned metrics while checking subgroup performance and deployment constraints.
How it connects
Connects performance evaluation with diagnostic evaluation, simulation, online testing, system latency, and LLM-as-a-judge.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.