Evaluation
Evaluation is the systematic assessment of a system, result, or approach against relevant criteria, evidence, or objectives.
Generative AI in Action
Evaluation is the systematic assessment of a model's generated outputs against specified criteria or requirements.
- How it is used
- Guides model selection, fine-tuning decisions, production monitoring, and assessment of application-specific harms.
- How deep it goes
- Develops criteria, baseline comparisons, metric interpretation, human-review procedures, and repeated harm checks without descending into mathematical derivations.
- Examples
- Compares article summaries with BLEU, ROUGE, BERTScore, and an LLM scoring coherence, consistency, fluency, and relevance.
- What you can do after
- Practice/perform task-specific evaluation by defining criteria, comparing baselines, interpreting complementary scores, and repeating checks after changes.
AI Agents in Action
Evaluation is the assessment of a solution, action, or outcome against criteria or expected results.
- How it is used
- Guides agent and assistant performance by checking responses, solutions, actions, or task success during or after interactions.
- How deep it goes
- Explains rubrics, similarity scores, automated LLM judges, and deployment placement, but stops short of a comprehensive metrics treatment.
- Examples
- Uses recommendation profiles, question answering, embedding similarity, scientific-article summaries, logical puzzles, and hypothetical experiments.
- What you can do after
- Evaluate/understand, compare prompt or profile variants, score outputs against rubrics or expected answers, and monitor agent performance.
Building LLM Applications with DSPy
The systematic assessment of an application’s or model’s outputs against specified quality criteria.
- How it is used
- Measures baselines and later versions to guide prompt optimization and judge likely production performance.
- How deep it goes
- Works through per-example metrics, aggregate scores, data splits, manual loops, and DSPy evaluation; defers prompt optimization itself.
- Examples
- Compares airline intent classifiers, a manual OpenAI prompt, and repeated LM and module trials with accuracy, duration, and exceptions.
- What you can do after
- Apply it via DSPy Evaluate to score a classifier, compare alternatives, and inform a production decision.
Read how Building LLM Applications with DSPy explains evaluation →
Machine Learning Platform Engineering
Model evaluation is the measurement of a trained model's performance on data separate from the data used for fitting.
- How it is used
- Uses evaluation to decide whether a trained model can advance or needs changes to its architecture or data.
- How deep it goes
- Details holdout curation, metric choice, automation, error analysis, and leakage risks, but does not give an evaluation implementation in the supplied excerpts.
- Examples
- Contrasts medical precision and false-positive concerns with financial recall, and contrasts exact-output tests with variable LLM responses.
- What you can do after
- Practice/perform repeatable holdout evaluation, investigate model errors, and select semantic rather than exact-match checks for variable LLM responses.
Read how Machine Learning Platform Engineering explains evaluation →
Introduction to Generative AI, Second Edition
Evaluation is the assessment of model outputs against specified criteria to determine how well a model performs a task.
- How it is used
- Uses task-specific assessments to compare prompts, models, and model versions before choosing changes for an application.
- How deep it goes
- Defines criteria and metrics, then outlines sample-input testing and prompt comparison without prescribing a scoring formula.
- Examples
- Uses property-listing descriptions, GAIA questions, and AssistantBench’s multi-website tasks to illustrate different evaluation needs.
- What you can do after
- Practice/perform it by defining task criteria, testing sample inputs, and comparing outputs while checking whether agent scores overlook risky behavior.
Read how Introduction to Generative AI, Second Edition explains evaluation →
Delivering Machine Learning Projects: From design to deployment
Performance evaluation is the systematic assessment of a model's predictive quality against defined data, metrics, and application requirements.
- How it is used
- Supports model selection by testing whether candidate performance meets application and business needs.
- How deep it goes
- Develops a staged process of baselines, candidate screening, and rigorous testing; leaves detailed performance-estimation methods for later treatment.
- Examples
- Uses a majority-class classifier and a customer-churn baseline to show why accuracy and small improvements need business context.
- What you can do after
- Practice/perform selective model evaluation and document why a chosen candidate is useful and verifiable.
Read how Delivering Machine Learning Projects: From design to deployment explains evaluation →
Reinforcement Learning from Human Feedback
Model evaluation is the systematic assessment of a model's behavior or performance against defined tasks, criteria, or benchmarks.
- How it is used
- Tracks post-training progress and measures targeted language-model capabilities through benchmarks.
- How deep it goes
- Explains benchmark regimes, prompt-dependent measurement, contamination, and cross-lab uncertainty without prescribing a universal evaluation stack.
- Examples
- Uses MMLU, GPQA, SWE-Bench, MATH, GSM8k, MT-Bench, AlpacaEval, and a formatted multiple-choice chain-of-thought prompt.
- What you can do after
- Evaluate/understand benchmark scores as configuration-sensitive signals rather than directly comparable rankings.
Read how Reinforcement Learning from Human Feedback explains evaluation →
Outlier Detection in Python
The assessment of a model or system against known outcomes, criteria, or reference results.
- How it is used
- Evaluates outlier detectors during model selection, tuning, follow-up, and—in ongoing systems—continued improvement.
- How deep it goes
- Explains labels, references, synthetic testing, score comparisons, and limitations, but stops short of prescribing one universal metric.
- Examples
- Uses labeled cases, injected synthetic outliers, cockpit dimensions, and baseball scores from KNN, Radius, and KDE detectors.
- What you can do after
- Diagnose detector performance and tune models, parameters, preprocessing, and data collection for the target outliers.
AI Agents in Action, Second Edition
Evaluation is the systematic assessment of an agent's output or behavior against defined criteria or expected results.
- How it is used
- Measures agent behavior against benchmarks to expose failures and guide improvements during development and deployment.
- How deep it goes
- Details benchmark design, repeated runs, holistic checks, and iteration; begins with string matching before broader rubric-based assessment.
- Examples
- Uses a RAG benchmark with expected and known-bad answers, followed by a string-matching check.
- What you can do after
- Practice/perform repeated benchmark checks, inspect intermediate behavior, and use failures to make minimal agent changes.
Read how AI Agents in Action, Second Edition explains evaluation →
Machine Learning System Design
Model evaluation is the process of measuring a machine learning model's performance using metrics on data separate from fitting data.
- How it is used
- Guides model comparison, deployment decisions, online experiments, and reporting against technical and business goals.
- How deep it goes
- Explains pipeline placement, holdout usage, metric selection, offline-online gaps, A/B testing, and reporting, without deriving metrics mathematically.
- Examples
- Uses training/validation/test splits, recommender feedback loops, prediction metrics, CTR, revenue, and A/B tests.
- What you can do after
- Evaluate models offline, protect test data, compare alternatives, design online tests, and report effects and limitations.
Read how Machine Learning System Design explains evaluation →
The Art of AI Product Development
Evaluation is the systematic assessment of a search system or model against specified criteria, data, or user outcomes.
- How it is used
- Guides release decisions and iterative improvement by testing search quality, generated responses, and real user outcomes.
- How deep it goes
- Moves from manual qualitative checks through quantitative metrics, component tests, and end-to-end evaluation with real-world data.
- Examples
- Uses workplace search queries, manual relevance ratings, task observation, retrieval metrics, fixed retrieval results, and real user responses.
- What you can do after
- Apply evaluation to identify improvement areas, compare system changes, and align performance with real user needs.
Read how The Art of AI Product Development explains evaluation →
Build an AI Agent (From Scratch)
Agent evaluation is the systematic measurement of an agent's behavior and task performance.
- How it is used
- Checks whether autonomous agents meet requirements without relying on a single successful final answer.
- How deep it goes
- Develops a repeatable process from execution records and error analysis through rubrics, metrics, test-set updates, and CI/CD checks.
- Examples
- Uses summarization requirements, search relevance, code-execution errors, and unnecessary tool calls to distinguish evaluation targets.
- What you can do after
- Practice/perform it by reviewing traces, codifying failures, automating checks, and updating tests as new failures emerge.
Read how Build an AI Agent (From Scratch) explains evaluation →
Build a Reasoning Model (From Scratch)
The process of assessing model performance on held-out data or benchmarks.
- How it is used
- Uses quantitative scoring to determine whether reasoning methods improve a pretrained LLM.
- How deep it goes
- Implements answer extraction, symbolic verification, dataset scoring, checkpoint evaluation, and validation-loss interpretation.
- Examples
- Uses MATH-500 accuracy, symbolic math verification, held-out validation sets, loss plots, and checkpoint metrics.
- What you can do after
- Diagnose model progress and compare reasoning improvements through verified accuracy, validation loss, and tracked training metrics.
Read how Build a Reasoning Model (From Scratch) explains evaluation →
Building Reliable AI Systems
The systematic measurement of an AI system's behavior and quality against defined criteria.
- How it is used
- Measures AI quality before deployment and over time so teams can identify failures and judge whether changes help.
- How deep it goes
- Outlines complementary methods, test-set coverage, CI/CD checks, and ongoing tracking without specifying a scoring rubric or thresholds.
- Examples
- Uses a chatbot’s post-deployment errors and a sample product query to illustrate monitoring needs and response inspection.
- What you can do after
- Practice/perform evaluation with varied test cases, complementary review methods, CI/CD checks, and production metric tracking.
AI-Powered Search
Evaluation is the systematic measurement of a model's performance on data separate from the examples used for training.
- How it is used
- Evaluates search-ranking models before deployment by measuring relevance on held-out queries.
- How deep it goes
- Provides an implementation-level split, training, ranking, and precision workflow, but defers broader metric selection and live validation.
- Examples
- Demonstrates query-level splitting, movie-ranking features, top-four precision near 0.3–0.4, and a test evaluation of 0.36.
- What you can do after
- Diagnose ranking quality with held-out queries, calculate precision, and identify features or examples needing improvement.
Related concepts
- metricuses · A metric is a quantitative measure used to characterize or assess a property, condition, behavior, performance, or outcome.
- computerized adaptive testkind of · A computerized adaptive test is an assessment that dynamically adjusts question selection or difficulty according to the test taker’s responses or performance.
- evaluation datasetuses · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
- language modelmeasures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.