liveBook · Manning

What you will learn in Evaluating AI Systems

Assessing effectiveness, reliability, and safety

by Panos Alexopoulos · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: The importance of AI evaluation

The chapter introduction uses real-world AI failures to show why evaluating complex systems before deployment matters and why it is difficult.

1.1 What is AI evaluation · 1.2 What is an AI system · 1.3 Why AI evaluation has become critical · 1.4 Why AI evaluation is challenging · 1.5 A systematic approach to AI evaluation · 1.6 The codex and the compass · 1.7 References · 1.8 Summary

What you learn in chapter 1 →

Chapter 2: Framing and scoping the evaluation

The introduction explains why an AI evaluation needs a shared purpose and scope before a team chooses how to measure it.

2.1 One room, many expectations · 2.2 Defining evaluation goals · 2.2.1 Setting expectations · 2.2.2 Comparing · 2.2.3 Deciding go/no-go · 2.2.4 Assessing risk · 2.2.5 Ensuring compliance · 2.2.6 Diagnosing and debugging · 2.2.7 Negotiating and prioritizing evaluation goals · 2.3 Determining the objects of the evaluation · 2.4 Defining the task to be evaluated · 2.4.1 Problems with AI task definitions · 2.4.2 Defining an AI task well · 2.5 Choosing evaluation dimensions · 2.6 References · 2.7 Summary

What you learn in chapter 2 →

Chapter 3: Picking evaluation metrics

The introduction explains how metrics turn evaluation goals into evidence and why choosing them requires more than picking a familiar measure.

3.1 “How do we measure this?” · 3.2 What is a metric · 3.3 What makes a good evaluation metric · 3.3.1 Construct validity · 3.3.2 Causal proximity · 3.3.3 Movability · 3.3.4 Reliability · 3.3.5 Robustness · 3.3.6 Correlation with human judgments · 3.3.7 Interpretability and explainability · 3.3.8 Measurability · 3.4 Designing custom metrics · 3.5 Managing and maintaining metrics · 3.6 References · 3.7 Summary

What you learn in chapter 3 →

Chapter 4: Acquiring evaluation data

This introduction explains what evaluation data can look like and why its suitability determines what an evaluation can reveal.

4.1 “What do we test it on?” · 4.2 Where to look for evaluation data · 4.2.1 Benchmarks · 4.2.2 Internal or operational data · 4.2.3 Real-world external data · 4.2.4 Human-authored data · 4.2.5 Synthetic data · 4.2.6 Choosing the right source of evaluation data · 4.3 Collecting and sampling evaluation data · 4.3.1 Sampling methods and strategies · 4.3.2 Determining how much evaluation data you need · 4.4 Synthesizing evaluation data · 4.4.1 Rule-based or template-based generation · 4.4.2 LLM-based data synthesis · 4.4.3 Handling the variety vs. correctness/plausibility trade-off · 4.4.4 Validating synthetic data against real data · 4.5 Assessing evaluation data quality · 4.5.1 Fitness to the evaluation task · 4.5.2 Accuracy and correctness · 4.5.3 Coverage and representativeness · 4.5.4 Absence of annotation artifacts and shortcuts · 4.6 References · 4.7 Summary

What you learn in chapter 4 →

Chapter 5: Acquiring judgments

This introduction defines judgments, explains their role in evaluation metrics, and shows why judgment quality affects evaluation results.

5.1 “Who and how decides what’s correct?” · 5.2 Deciding what judgments should look like · 5.2.1 Binary vs. scalar judgments · 5.2.2 Absolute vs. relative judgments · 5.3 Defining rubrics and providing guidelines · 5.3.1 Breaking constructs into observable criteria · 5.3.2 Giving examples and counterexamples · 5.3.3 Writing instructions that minimize ambiguity and vagueness · 5.4 Deciding who will judge · 5.4.1 Domain experts · 5.4.2 Trained non-experts · 5.4.3 Crowdsourced workers · 5.4.4 AI-as-a-Judge · 5.5 Calibrating human judges · 5.6 Developing and validating LLM judges · 5.6.1 Which LLMs to use as judges · 5.6.2 Writing effective judgment prompts · 5.6.3 Validating and improving LLM judges · 5.6.4 Case study: Validating LLM judges for RAG faithfulness evaluation · 5.7 Designing judgment workflows · 5.7.1 Number and allocation of judges · 5.7.2 Order and sequence of judgments · 5.7.3 Judgment aggregations · 5.7.4 Judgment quality monitoring · 5.8 References · 5.9 Summary

What you learn in chapter 5 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.