Evaluation
Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
Generative AI in Action
Evaluation is the systematic assessment of a model's generated outputs against specified criteria or requirements.
- How it is used
- Guides model selection, fine-tuning decisions, and ongoing assessment of production applications against task and safety requirements.
- How deep it goes
- Develops an evaluation cycle with criteria, baselines, metrics, human review, and repeated testing; outlines DeepEval without detailed coverage.
- Examples
- Compares article summaries using reference-based scores and an LLM judge with explicit criteria.
- What you can do after
- Apply it via Hugging Face Evaluate to compare summaries using BLEU, ROUGE, and BERTScore.
LLM Evaluation and Alignment, The Foundational Ideas
Evaluation is the systematic measurement of whether a model, agent, or AI system performs a requested task accurately and reliably, including on edge cases.
- How it is used
- Measures task performance and edge cases to guide iterative alignment and production monitoring.
- How deep it goes
- Explains task classification, metric selection, error analysis, and repeated evaluation, with trace-level diagnosis and implementation-oriented methods.
- Examples
- Uses customer service conversations, missing shipping data, and legal research agent traces to expose distinct failure modes.
- What you can do after
- Diagnose/debug AI systems by selecting criteria, analyzing failures, and re-evaluating after changes.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains evaluation →
Enterprise RAG
Evaluation is the systematic assessment of a system's behavior, quality, or performance against defined criteria.
- How it is used
- Defines RAG chatbot scope as stakeholder-approved question-and-answer test cases before implementation.
- How deep it goes
- Details an end-to-end workflow from eval design through LLM-based comparisons, pytest assertions, and CI automation.
- Examples
- Uses product facts, customer order history, and product-database availability as evaluation cases.
- What you can do after
- Build it from scratch as automated RAG tests that compare generated answers with expected answers and detect regressions.
AI Model Evaluation
A performance evaluation measures how well a model or AI system achieves its intended objective against predefined metrics.
- How it is used
- Uses offline and online evaluation to assess model quality and product impact before and after user exposure.
- How deep it goes
- Defines performance evaluation, compares complementary methods, and gives metric, diagnostic, and infrastructure guidance without treating scores as explanations.
- Examples
- Calculates movie recommendation recall and examines how aggregate results can conceal differences between user groups.
- What you can do after
- Evaluate model versions against product-aligned metrics while checking subgroup performance and deployment constraints.
AI Agents in Action
Evaluation is the assessment of a solution, action, or outcome against criteria or expected results.
- How it is used
- Guides agent performance and compares responses, plans, and profile variations against criteria or expected results.
- How deep it goes
- Works through rubric scoring and embedding similarity flows, and surveys internal and external evaluation across agent applications.
- Examples
- Uses scored recommendations, expected-answer similarity, and an exercise evaluating an experimental prediction.
- What you can do after
- Apply it via prompt flow to compare profile variations or evaluate question-answering responses.
Building LLM Applications with DSPy
The systematic assessment of an application’s or model’s outputs against specified quality criteria.
- How it is used
- Uses evaluation to compare baseline and improved LM applications and assess production readiness.
- How deep it goes
- Works through per-response metrics, aggregate scoring, manual and DSPy evaluation, repeated trials, and final-evaluation trade-offs; defers optimization.
- Examples
- Evaluates an airline intent classifier and compares model and module configurations across accuracy, duration, and exceptions.
- What you can do after
- Apply it via DSPy Evaluate or a custom Python loop to score examples and compare application configurations.
Read how Building LLM Applications with DSPy explains evaluation →
Machine Learning Platform Engineering
Model evaluation is the measurement of a trained model's performance on data separate from the data used for fitting.
- How it is used
- Uses evaluation to decide whether a trained model can advance or needs changes to its architecture or data.
- How deep it goes
- Details holdout curation, metric choice, automation, error analysis, and leakage risks, but does not give an evaluation implementation in the supplied excerpts.
- Examples
- Contrasts medical precision and false-positive concerns with financial recall, and contrasts exact-output tests with variable LLM responses.
- What you can do after
- Practice/perform repeatable holdout evaluation, investigate model errors, and select semantic rather than exact-match checks for variable LLM responses.
Read how Machine Learning Platform Engineering explains evaluation →
Introduction to Generative AI, Second Edition
Evaluation is the assessment of model outputs against specified criteria to determine how well a model performs a task.
- How it is used
- Uses evaluation to compare prompt and model changes and assess whether agents are reliable enough to deploy.
- How deep it goes
- Defines metrics and outlines criteria, sample collection, and scoring, while noting that answer-focused agent benchmarks can overlook process.
- Examples
- Uses property listing descriptions and the GAIA agent benchmark to illustrate task criteria and benchmark limitations.
- What you can do after
- Apply evaluation by defining task criteria, testing representative inputs, and comparing scored responses across prompt changes.
Read how Introduction to Generative AI, Second Edition explains evaluation →
Delivering Machine Learning Projects: From design to deployment
Performance evaluation is the systematic assessment of a model's predictive quality against defined data, metrics, and application requirements.
- How it is used
- Supports selecting models that meet application requirements and inspire deployment confidence.
- How deep it goes
- Outlines baselining, selective full evaluation, test-environment considerations, and documentation, while deferring detailed performance-estimation methods.
- Examples
- Uses a 99%-accurate majority-class classifier and a customer-churn baseline to illustrate contextual performance and business value.
- What you can do after
- Evaluate model candidates against baselines and business requirements, and plan selective testing with documented evidence.
Read how Delivering Machine Learning Projects: From design to deployment explains evaluation →
Context Engineering
Evaluation is the structured testing and measurement of an AI system's behavior and performance.
- How it is used
- Provides a repeatable feedback loop for assessing context-aware AI behavior and guiding development decisions.
- How deep it goes
- Defines eval components and grader trade-offs, then applies targeted probes and a library-based retrieval evaluation.
- Examples
- Uses debugging-session recall probes, coding-agent artifact checks, and a RAG scoring example.
- What you can do after
- Apply evaluation via Ragas to score retrieval and generation, calibrating model judgments before release use.
Decision Engineering
The decision activity of considering alternatives and converging on a choice.
- How it is used
- Guides the comparison of alternatives after problem framing and before a technical choice is put into practice.
- How deep it goes
- Provides a staged external-technology evaluation procedure, from screening to focused trials and commercial checks, rather than a universal scoring formula.
- Examples
- Uses a visual-editor trial, comparable technology prototypes, and vendor pricing or data questions to expose gaps in paper comparisons.
- What you can do after
- Practice/perform a stakes-calibrated technology comparison that tests decisive concerns before committing to a candidate.
Reinforcement Learning from Human Feedback
Model evaluation is the systematic assessment of a model's behavior or performance against defined tasks, criteria, or benchmarks.
- How it is used
- Uses benchmarks and feedback to track model quality and guide post-training decisions.
- How deep it goes
- Explains prompting, scoring, comparison variability, held-out testing, and contamination, without prescribing a universal evaluation workflow.
- Examples
- Uses MMLU’s multiple-choice reasoning prompt, chat benchmarks with LLM judges, and cross-lab score variability to illustrate evaluation choices.
- What you can do after
- Evaluate and understand benchmark results by accounting for prompt format, sampling variability, test status, and contamination.
Read how Reinforcement Learning from Human Feedback explains evaluation →
Outlier Detection in Python
The assessment of a model or system against known outcomes, criteria, or reference results.
- How it is used
- Evaluates outlier detectors during model selection, tuning, follow-up, and—in ongoing systems—continued improvement.
- How deep it goes
- Explains labels, references, synthetic testing, score comparisons, and limitations, but stops short of prescribing one universal metric.
- Examples
- Uses labeled cases, injected synthetic outliers, cockpit dimensions, and baseball scores from KNN, Radius, and KDE detectors.
- What you can do after
- Diagnose detector performance and tune models, parameters, preprocessing, and data collection for the target outliers.
AI Agents in Action, Second Edition
Evaluation is the systematic assessment of an agent's output or behavior against defined criteria or expected results.
- How it is used
- Uses evaluation and feedback to make agent robustness measurable and guide iterative improvement.
- How deep it goes
- Details benchmark setup, holistic run checks, repeated testing, and incremental changes, then extends evaluation into automated checks and release gates.
- Examples
- Tests a fictional knowledge agent with expected and known-bad answers, initially matching expected strings.
- What you can do after
- Practice test-driven agent development by defining benchmarks, repeating tests, and making minimal prompt or tool changes.
Read how AI Agents in Action, Second Edition explains evaluation →
Machine Learning System Design
Model evaluation is the process of measuring a machine learning model's performance using metrics on data separate from fitting data.
- How it is used
- Guides model comparison, deployment decisions, online experiments, and reporting against technical and business goals.
- How deep it goes
- Explains pipeline placement, holdout usage, metric selection, offline-online gaps, A/B testing, and reporting, without deriving metrics mathematically.
- Examples
- Uses training/validation/test splits, recommender feedback loops, prediction metrics, CTR, revenue, and A/B tests.
- What you can do after
- Evaluate models offline, protect test data, compare alternatives, design online tests, and report effects and limitations.
Read how Machine Learning System Design explains evaluation →
The Art of AI Product Development
Evaluation is the systematic assessment of a search system or model against specified criteria, data, or user outcomes.
- How it is used
- Guides release decisions and iterative improvement by testing search quality, generated responses, and real user outcomes.
- How deep it goes
- Moves from manual qualitative checks through quantitative metrics, component tests, and end-to-end evaluation with real-world data.
- Examples
- Uses workplace search queries, manual relevance ratings, task observation, retrieval metrics, fixed retrieval results, and real user responses.
- What you can do after
- Apply evaluation to identify improvement areas, compare system changes, and align performance with real user needs.
Read how The Art of AI Product Development explains evaluation →
Applied Reinforcement Learning
The assessment of a system's performance using evidence and criteria intended to distinguish effective behavior from misleading results.
- How it is used
- Tests whether an RL system’s apparent performance reflects skill or luck, and helps explain why a change improved results.
- How deep it goes
- Specifies held-out testing, controlled comparisons, aggregation, risk-adjusted metrics, and signal-fidelity checks, without detailing statistical tests.
- Examples
- Uses portfolio modes with different signals, shared market conditions, and an oracle comparison; identical outputs expose possible signal neglect.
- What you can do after
- Diagnose whether an agent uses its inputs and compare performance under controlled, risk-aware tests.
Read how Applied Reinforcement Learning explains evaluation →
Machines that Think
Evaluation is the systematic assessment of a model's outputs or capabilities against specified criteria or reference results.
- How it is used
- Uses evaluation to guide trust, verification, and oversight of model outputs in practical contexts.
- How deep it goes
- Compares benchmark goals and examines how puzzle design, output requirements, and interfaces complicate interpretation of failures.
- Examples
- Illustrates evaluation with MMLU, GSM8K, SWE-bench, Chatbot Arena, Tower of Hanoi, and River Crossing.
- What you can do after
- Evaluate model results by distinguishing observed performance from the effects of scoring, task design, and available tools.
Build an AI Agent (From Scratch)
Agent evaluation is the systematic measurement of an agent's behavior and task performance.
- How it is used
- Uses automated evaluation to check autonomous agent quality and prevent regressions as the system evolves.
- How deep it goes
- Organizes evaluation around outputs, components, and trajectories, then outlines trace review, failure taxonomies, rubrics, metrics, and test-set updates.
- Examples
- Uses summarization accuracy and format, search relevance, code execution errors, and unnecessary tool calls to illustrate evaluation targets.
- What you can do after
- Practice evaluating agent outputs and execution paths, then use observed failures to refine quality checks.
Read how Build an AI Agent (From Scratch) explains evaluation →
Build a Reasoning Model (From Scratch)
The process of assessing model performance on held-out data or benchmarks.
- How it is used
- Uses quantitative scoring to determine whether reasoning methods improve a pretrained LLM.
- How deep it goes
- Implements answer extraction, symbolic verification, dataset scoring, checkpoint evaluation, and validation-loss interpretation.
- Examples
- Uses MATH-500 accuracy, symbolic math verification, held-out validation sets, loss plots, and checkpoint metrics.
- What you can do after
- Diagnose model progress and compare reasoning improvements through verified accuracy, validation loss, and tracked training metrics.
Read how Build a Reasoning Model (From Scratch) explains evaluation →
Vibe Engineering
The assessment of an artifact or system to determine its quality, correctness, or fitness against relevant criteria.
- How it is used
- Uses test-set metrics to assess price predictions and frames code verification as a constraint on trusting generated code.
- How deep it goes
- Defines three regression metrics and test-set comparison, but treats code verification as a capacity problem rather than a procedural checklist.
- Examples
- Uses a drug-price prediction pipeline, held-out test data, and R², MAE, and RMSE as concrete evaluation examples.
- What you can do after
- Evaluate price-prediction models by comparing test-set predictions with actual prices and interpreting three complementary error metrics.
Designing AI Agents
Evaluation is the systematic assessment of a result against criteria for quality, correctness, or suitability.
- How it is used
- Places evaluation beside agent changes to decide whether artifacts, recoveries, skills, and lessons are fit for use.
- How deep it goes
- Explains evidence selection and limitations across reflection patterns, including admission checks, delayed outcomes, and cross-run review.
- Examples
- Uses accumulated build safeguards and golden SQL comparisons to illustrate system-level costs and production failure localization.
- What you can do after
- Diagnose failures with traces and evaluation results, then use localized evidence to guide architecture changes.
Software Engineering for Data Scientists
Model evaluation is the assessment of a predictive model's performance on data, typically including data not used for development.
- How it is used
- Checks whether trained models perform reliably on fresh data before and during ongoing prediction use.
- How deep it goes
- Defines hold-out evaluation and several metrics, then extends the practice to ground-truth comparisons and production monitoring.
- Examples
- Uses hold-out datasets, continuous predictions, and customer churn cases to demonstrate metrics and ground-truth comparisons.
- What you can do after
- Diagnose model performance with task-appropriate metrics, ground-truth comparisons, dashboards, and performance alerts.
Read how Software Engineering for Data Scientists explains evaluation →
Retrieval Augmented Generation, The Foundational Ideas
Systematic evaluation is a structured process for measuring system behavior against defined criteria and test conditions.
- How it is used
- Diagnoses RAG reliability problems before production and supports decisions about whether added complexity delivers measurable value.
- How deep it goes
- Explains three reference-free metrics and targeted failure tests, but stops short of specifying metric formulas or complete test implementations.
- Examples
- Uses faithfulness, answer relevance, context relevance, failure-point test suites, and an agentic assistant accuracy-latency comparison.
- What you can do after
- Diagnose/debug with targeted metrics and test suites, then compare accuracy gains against latency and complexity costs.
Read how Retrieval Augmented Generation, The Foundational Ideas explains evaluation →
Building Reliable AI Systems
The systematic measurement of an AI system's behavior and quality against defined criteria.
- How it is used
- Guides reliability work from defining success before development through monitoring behavior after deployment.
- How deep it goes
- Outlines an evaluation workflow and complementary methods without specifying metric calculations or acceptance thresholds.
- Examples
- Uses a wireless-headphone chatbot query and a scenario involving subtly wrong answers or demographic disparities.
- What you can do after
- Evaluate and understand AI reliability by combining assessment methods, tracking quality, and checking for regressions.
Build AI Drug Discovery Pipelines
Model evaluation is the assessment and comparison of model predictions, including analysis of predictive errors.
- How it is used
- Guides comparison of predictive models, error analysis, iteration, and decisions about whether a model meets its practical objective.
- How deep it goes
- Demonstrates train-set accuracy and discusses test data, confusion-matrix metrics, cross-validation, overfitting, and regularization without covering deployment maintenance.
- Examples
- Uses molecular fingerprints, a stochastic-gradient-descent classifier, hERG blocker labels, a train-test split, and a 0.964 training accuracy calculation.
- What you can do after
- Diagnose model quality with held-out data, classification metrics, cross-validation, and error analysis while recognizing overfitting.
Read how Build AI Drug Discovery Pipelines explains evaluation →
Designing AI Systems
Evaluation is the systematic assessment of an AI system or change against defined evidence and criteria.
- How it is used
- Uses evaluation to decide whether changes to prompts, models, or retrieval parameters deserve promotion to production.
- How deep it goes
- Defines three evaluation modes and their roles, then illustrates dataset-based prompt comparison without detailing scoring algorithms.
- Examples
- Compares formal and conversational patient-intake prompts using questions, ideal responses, and key elements as test data.
- What you can do after
- Evaluate prompt or configuration changes against shared benchmarks and use results to guide improvement or promotion.
AI-Powered Search
Evaluation is the systematic measurement of a model's performance on data separate from the examples used for training.
- How it is used
- Evaluates search-ranking models before deployment by measuring relevance on held-out queries.
- How deep it goes
- Provides an implementation-level split, training, ranking, and precision workflow, but defers broader metric selection and live validation.
- Examples
- Demonstrates query-level splitting, movie-ranking features, top-four precision near 0.3–0.4, and a test evaluation of 0.36.
- What you can do after
- Diagnose ranking quality with held-out queries, calculate precision, and identify features or examples needing improvement.
AI Recommender Systems
Evaluation is the systematic assessment of a system or approach by comparing its decisions or outcomes against a reference, alternative, or objective.
- How it is used
- Frames evaluation as a communication tool for comparing recommendation approaches and deciding whether changes are worth shipping.
- How deep it goes
- Develops a practical workflow covering splits, baselines, metrics, sampling, spot checks, pipeline stages, and user lifecycle gaps.
- Examples
- Uses random, popularity, and personalized baselines, staged metrics, sampled users, capped ratings, and Star Wars recommendation spot checks.
- What you can do after
- Apply it via comparative metrics, temporal splits, baseline ladders, sampled runs, spot checks, and lifecycle-aware analysis.
Also covered in
- Investing for Programmers
- The Architecture of ReasoningIn early access (MEAP)
- Knowledge Graphs and LLMs in Action
- Build an Advanced RAG Application (From Scratch)In early access (MEAP)
- Fundamentals of CybersecurityIn early access (MEAP)
- Grokking Statistics
- Self-Improving AgentsIn early access (MEAP)
- Fight Fraud with Machine LearningIn early access (MEAP)
- Think Like a CTO
- Data-Driven Cybersecurity
- Building Specialized AI SystemsIn early access (MEAP)
- Deep Learning with R, Third Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- AI agentAn AI agent is a computational system that uses artificial intelligence to interpret information and select or perform actions toward goals while interacting with an environment.
Related concepts
- metricuses · A metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.
- artificial intelligence systemmeasures · An artificial intelligence system is a computational system that uses artificial intelligence techniques to perform tasks such as learning, reasoning, interpreting, generating, predicting, or making decisions.
- computerized adaptive testkind of · A computerized adaptive test is an assessment that dynamically adjusts question selection or difficulty according to the test taker’s responses or performance.
- decision lifecyclepart of · A model of decision-making as a set of recurring activities that move from clarifying a problem through evaluating options to carrying out a choice.
- drawdownuses · A drawdown is a decline in an investment’s or market’s value from a prior peak to a subsequent low.
- Sharpe ratiouses · The Sharpe ratio measures risk-adjusted investment return as excess return over the risk-free rate per unit of return volatility.
Questions about evaluation
What is evaluation?
Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
Which Manning books explain evaluation?
Generative AI in Action, LLM Evaluation and Alignment, The Foundational Ideas, Enterprise RAG, AI Model Evaluation, AI Agents in Action, Building LLM Applications with DSPy, Machine Learning Platform Engineering, Introduction to Generative AI, Second Edition and 21 more.
What should I know before learning evaluation?
Manning books that teach evaluation usually introduce language model, accuracy, retrieval-augmented generation and AI agent first.
What is evaluation a kind of?
Evaluation is a kind of computerized adaptive test. A computerized adaptive test is an assessment that dynamically adjusts question selection or difficulty according to the test taker’s responses or performance.
What is evaluation part of?
Evaluation is part of decision lifecycle. A model of decision-making as a set of recurring activities that move from clarifying a problem through evaluating options to carrying out a choice.
What is evaluation also called?
Evaluation is also called evaluating, evaluate, assessment or evals.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




























