Chapter 1: The importance of AI evaluation
The chapter introduction uses real-world AI failures to show why evaluating complex systems before deployment matters and why it is difficult.
After this chapter you will be able to
- Predeployment Evaluation Challenge. Explain why the incidents illustrate a fundamental challenge for AI evaluation: complex systems can fail when exposed to the full variability of the real world, even if risks were considered beforehand. State why the specific causes cannot be established from these accounts alone.
- Real-world evaluation purpose. Explain what AI evaluation measures and why model performance or benchmark scores alone are insufficient.
- Breadth of evaluation questions. Describe the kinds of questions AI evaluation asks beyond accuracy, and explain why they call for a structured approach grounded in context, purpose, and impact.
- Task Based AI Definition. Explain how the section defines an AI system and why evaluation emphasizes what it does, how well it does it, and whether its behavior is acceptable in context rather than its underlying technology.
- Model Versus Complete System. Distinguish an AI model from the complete AI system and justify evaluating the system users actually encounter.
- System Categories And Overlap. Distinguish predictive, generative, and agentic systems by what they do, and explain why overlapping categories make the intended task a better starting point for evaluation than the AI technique.
- Balanced deployment through evaluation. Distinguish over-reliance from under-reliance on AI and explain why rigorous, continuous evaluation is needed for evidence-based decisions about where, how, and whether to deploy it.
- The AI evaluation gap. Explain the AI evaluation gap as the mismatch between modern AI capabilities and the practices used to assess them.
- Shallow and delayed evaluation. Describe how narrow benchmark checks, late evaluation, and unclear ownership can leave consequential failures undiscovered until deployment.
- Unpredictable AI system behavior. Distinguish AI evaluation from testing conventional deterministic software by explaining why AI behavior is harder to specify and predict.
- Multidimensional evaluation needs. Explain why evaluating accuracy or correctness alone is insufficient for an AI system.
- Evaluation frame and scope. Explain why an evaluation must establish its purpose, object, tasks, and quality dimensions before testing. Use the legal chatbot to illustrate how testing only its language model or factual accuracy could miss important behavior of the system users encounter.
- Ordered evaluation activities. Trace the six activities from framing and scoping through selecting metrics, acquiring data, computing metrics, interpreting and communicating results, and deciding next steps. Explain why metrics should reflect the evaluation's aims, data should fit the tasks and real-world context, and computation should be dependable.
- Results and next decisions. Explain why raw scores require interpretation for particular users, uses, and risks, and how findings should be communicated to stakeholders and lead to a decision about next steps.
- Iteration and human oversight. Explain why the evaluation activities depend on one another and should be repeated across the AI development lifecycle, including before and during deployment. Distinguish activities automation can accelerate from judgments that still require human oversight.
Section by section
Introduction
The chapter introduction uses real-world AI failures to show why evaluating complex systems before deployment matters and why it is difficult.
- Predeployment Evaluation Challenge: Explain why the incidents illustrate a fundamental challenge for AI evaluation: complex systems can fail when exposed to the full variability of the real world, even if risks were considered beforehand. State why the specific causes cannot be established from these accounts alone.
Why it matters: It explains why predeployment evaluation is important without assuming every failure was foreseeable or negligent. - Varied Consequences of AI Failures: Describe how the examples show different serious consequences of AI systems failing after deployment, such as financial losses, discriminatory screening, and incorrect customer advice.
Why it matters: The examples make the stakes of evaluation concrete across different applications.
Key terms: AI systems, after deployment, full variability of the real world, postmortem, Zillow Offers, AI-powered recruiting software, customer-service chatbot
1.1 What is AI evaluation
Defines AI evaluation as a systematic assessment of how reliably an AI system serves its intended purpose in real-world use, beyond model scores alone.
- Real-world evaluation purpose: Explain what AI evaluation measures and why model performance or benchmark scores alone are insufficient.
Why it matters: It keeps an evaluation focused on whether the system works for its intended use, not just whether it scores well on a test. - Breadth of evaluation questions: Describe the kinds of questions AI evaluation asks beyond accuracy, and explain why they call for a structured approach grounded in context, purpose, and impact.
Why it matters: These questions reveal consequential behavior that an accuracy number alone can miss.
Key terms: AI evaluation, intended context of use, intended purpose, user groups and contexts, edge cases, failure modes, blind spots, risks
1.2 What is an AI system
This section defines AI systems by the tasks they perform, distinguishes complete systems from models, and explains why evaluation should focus on system behavior in context.
- Task Based AI Definition: Explain how the section defines an AI system and why evaluation emphasizes what it does, how well it does it, and whether its behavior is acceptable in context rather than its underlying technology.
Why it matters: This definition keeps the evaluation focused on the work the system is meant to do, even when its technology changes. - Hybrid Components In Practice: Describe how the CV parsing example combines machine learning with a knowledge graph and why both matter to its task.
Why it matters: The example shows why evaluating only a learned model could miss a component important to the system’s results. - Model Versus Complete System: Distinguish an AI model from the complete AI system and justify evaluating the system users actually encounter.
Why it matters: A model score alone may not reflect whether the product is reliable, safe, or useful to its users. - System Categories And Overlap: Distinguish predictive, generative, and agentic systems by what they do, and explain why overlapping categories make the intended task a better starting point for evaluation than the AI technique.
Why it matters: The categories help describe a system, but their overlap means its actual task is more informative for evaluation.
Key terms: AI system, human intelligence tasks, context, hybrid AI, machine learning, knowledge graph, CV parsing, AI model, data pipelines, human-in-the-loop mechanisms
1.3 Why AI evaluation has become critical
The section explains why expanding AI capabilities and persistent uncertainty make rigorous, continuous evaluation necessary for sound deployment decisions.
- Expanded capabilities and consequences: Explain how AI’s move beyond narrow, task-specific models has made evaluation more important as systems take on more kinds of tasks with material and long-term consequences.
Why it matters: Evaluation matters more when AI outputs can affect consequential decisions and activities. - Complexity and comprehension gap: Describe why increasingly capable AI systems can remain difficult to predict, control, or explain, and what teams may not know about their performance and risks.
Why it matters: Recognizing this gap explains why apparent capability alone is not enough to guide deployment. - Balanced deployment through evaluation: Distinguish over-reliance from under-reliance on AI and explain why rigorous, continuous evaluation is needed for evidence-based decisions about where, how, and whether to deploy it.
Why it matters: Evaluation helps teams avoid both preventable harm and missed benefits.
Key terms: narrow, task-specific models, material and long-term consequences, complexity, gap between capability and comprehension, over-reliance, under-reliance, rigorous and continuous evaluation, evidence-based decisions
1.4 Why AI evaluation is challenging
This section explains why current evaluation practices often miss consequential AI failures and why assessing AI systems is inherently difficult.
- The AI evaluation gap: Explain the AI evaluation gap as the mismatch between modern AI capabilities and the practices used to assess them.
Why it matters: The gap explains why familiar evaluation habits may not give enough confidence in an AI system. - Shallow and delayed evaluation: Describe how narrow benchmark checks, late evaluation, and unclear ownership can leave consequential failures undiscovered until deployment.
Why it matters: A system can look good on a benchmark yet fail in circumstances that matter after release. - Unpredictable AI system behavior: Distinguish AI evaluation from testing conventional deterministic software by explaining why AI behavior is harder to specify and predict.
Why it matters: A test that passes in development may not reveal how an AI system will behave in production. - Multidimensional evaluation needs: Explain why evaluating accuracy or correctness alone is insufficient for an AI system.
Why it matters: A correct output does not by itself show that a system is suitable for people or its deployment setting.
Key terms: AI evaluation gap, standard metrics, public benchmark dataset, fit for purpose, deterministic, probabilistically, deployment context, edge cases, multidimensionality, robust
1.5 A systematic approach to AI evaluation
The section explains how to evaluate an AI system through an ordered, iterative process rather than by running convenient metrics and reporting their scores.
- Evaluation frame and scope: Explain why an evaluation must establish its purpose, object, tasks, and quality dimensions before testing. Use the legal chatbot to illustrate how testing only its language model or factual accuracy could miss important behavior of the system users encounter.
Why it matters: A well-chosen metric cannot rescue an evaluation aimed at the wrong system behavior. - Ordered evaluation activities: Trace the six activities from framing and scoping through selecting metrics, acquiring data, computing metrics, interpreting and communicating results, and deciding next steps. Explain why metrics should reflect the evaluation's aims, data should fit the tasks and real-world context, and computation should be dependable.
Why it matters: The order helps ensure that collected data and calculated scores answer the question the evaluation was meant to ask. - Results and next decisions: Explain why raw scores require interpretation for particular users, uses, and risks, and how findings should be communicated to stakeholders and lead to a decision about next steps.
Why it matters: Without context, stakeholders may treat a favorable score as evidence for a conclusion it does not support. - Iteration and human oversight: Explain why the evaluation activities depend on one another and should be repeated across the AI development lifecycle, including before and during deployment. Distinguish activities automation can accelerate from judgments that still require human oversight.
Why it matters: Repeated evaluation and human review help keep the evidence relevant as system behavior and user needs change.
Key terms: evaluation goals, objects, tasks, quality dimensions, frame and scope, evaluation metrics, evaluation data, compute the metrics, interpret and communicate, next steps
1.6 The codex and the compass
This section describes the book’s practical approach, supporting materials, and recommended background without teaching an AI evaluation concept.
1.7 References
This section lists sources cited in the chapter without explaining new concepts.
1.8 Summary
Where this is used later
- Defining the task to be evaluated (chapter 2) · Defining the task depends on identifying the purpose and use being evaluated.
- Choosing evaluation dimensions (chapter 2) · Choosing evaluation dimensions builds on the range of questions an evaluation may need to address.
- Determining the objects of the evaluation (chapter 2) · The idea that an AI system can include multiple components.
- Defining evaluation goals (chapter 2) · Motivates defining what an evaluation must help decision-makers decide.
- What is a metric (chapter 3) · Examines the metrics selected after framing and scoping.
- What makes a good evaluation metric (chapter 3) · Develops criteria for choosing useful evaluation metrics.
Concepts explained in this chapter
- evaluation activityA step in the process of planning, executing, interpreting, or acting on an AI evaluation.
- artificial intelligence systemA system incorporating artificial intelligence whose real-world impact can depend on models, interfaces, data pipelines, human interaction, and policies.
- evaluation datasetData on which an AI system is assessed and from which evaluation measures are computed.
- metricA quantitative or scored measure used to assess a system against an evaluation criterion.
- evaluation criterionA stated condition or standard used to determine whether a system satisfies a decision requirement.
- evaluation goalThe intended purpose or decision that an evaluation is meant to support.
- evaluation objectThe AI system, subsystem, or component whose performance or properties are being assessed.
- evaluation taskAn evaluation task is an activity an AI system is expected to perform and whose outcomes can be assessed.
- interpretabilityThe extent to which a system’s behavior, outputs, or evaluation measures can be made understandable to people.
- robustnessThe ability of a system to maintain suitable behavior under varied, unclear, or challenging inputs and conditions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
