liveBook · Manning

Chapter 4: Acquiring evaluation data

What you will learn in chapter 4 of Evaluating AI Systems

This introduction explains what evaluation data can look like and why its suitability determines what an evaluation can reveal.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains what evaluation data can look like and why its suitability determines what an evaluation can reveal.

Key terms: evaluation data, inputs, examples, test cases, metrics, subgroups, dangerous edge cases, behaviors, risks, capabilities

4.1 “What do we test it on?”

The section uses a medical chatbot scenario to illustrate trade-offs in choosing evaluation data, sampling cases, and using synthetic examples.

Key terms: public datasets, benchmarks, logs, real usage, sensitive information, anonymize, legal basis, purpose limitation, patient privacy, sampling

4.2 Where to look for evaluation data

The section introduces evaluation data sources and the trade-offs involved in choosing among them.

Key terms: evaluation data sources, advantages, risks, costs

4.2.1 Benchmarks

This section explains what benchmark datasets offer, when they are useful or misleading, and how annotation artifacts can inflate apparent performance.

Key terms: benchmark datasets, reference outputs, reproducible evaluations, baseline analysis, academic repositories, community leaderboards, licensing, representativeness, freshness, difficulty

4.2.2 Internal or operational data

This section describes internal operational data, where to find it, and when its realism or limitations make it suitable for evaluation.

Key terms: internal data, operational data, benchmarks, production logs, organizational databases, internal monitoring pipelines, observability tools, data warehouses, practical effectiveness, user alignment

4.2.3 Real-world external data

This section explains what external real-world evaluation data is and when its breadth is useful or its lack of deployment-specific alignment is limiting.

Key terms: external real-world datasets, benchmark datasets, generalization, stress-testing, domain-shift evaluation, fairness assessments, robustness checks, deployment context

4.2.4 Human-authored data

This section explains human-authored evaluation data and when its control and flexibility outweigh its limitations.

Key terms: human-authored data, inputs, labels, judgments, controlled conditions, failure modes, edge cases, representativeness, production monitoring, drift detection

4.2.5 Synthetic data

This section defines synthetic evaluation data and explains when it is useful and why it cannot replace real-world evidence.

Key terms: synthetic evaluation data, templated prompts, simulated environments, synthetic data, stress testing, operational data, realistic user behavior, natural settings, ambiguity, diversity

4.2.6 Choosing the right source of evaluation data

This section explains how to combine evaluation data sources and acquire data responsibly under practical, legal, and ethical constraints.

Key terms: benchmarks, internal operational data, synthetic data, human-generated data, legal compliance, GDPR, HIPAA, data-processing agreements, privacy impact assessments, data minimization

4.3 Collecting and sampling evaluation data

This section introduces the need to make deliberate choices when collecting or sampling evaluation data.

4.3.1 Sampling methods and strategies

This section compares sampling strategies for estimating performance, analyzing subgroups, finding weaknesses, and detecting changes across conditions.

Key terms: random sampling, overall system performance, stratified sampling, strata, subgroup performance, adversarial sampling, stress testing, risk discovery, error analysis, observed failures

4.3.2 Determining how much evaluation data you need

This section explains how evaluation goals, task variability, metric stability, and metric movability help determine whether an evaluation dataset is sufficient.

Key terms: evaluation goal, quick diagnostics, benchmark-style comparisons, risk assessment, task variability, task complexity, coverage, subgroups, metric stability, confidence intervals

4.4 Synthesizing evaluation data

This section introduces rule-based and LLM-based approaches to generating synthetic evaluation data.

4.4.1 Rule-based or template-based generation

This section explains how rule-based templates generate controlled synthetic evaluation data and where that control is useful in testing.

Key terms: rule-based generation, synthetic evaluation data, templates, logical constraints, placeholders, controlled vocabularies, structured domain lists, randomization, deterministic logic, expected outputs

4.4.2 LLM-based data synthesis

This section explains how to prompt an LLM to generate diverse evaluation examples and iteratively review and improve them.

Key terms: LLM-based data synthesis, synthetic examples, rule-based generation, base prompt, constraints, realistic patient questions, examples, topic lists, external materials, stylistic and difficulty controls

4.4.3 Handling the variety vs. correctness/plausibility trade-off

This section explains how the consequences of evaluation mistakes guide the balance between variety and correctness or plausibility in synthetic data.

Key terms: variety, correctness, plausibility, false alarm

4.4.4 Validating synthetic data against real data

This section explains how to validate synthetic evaluation data against real data by comparing characteristics, failure modes, and difficulty.

Key terms: distributions, characteristics, topical coverage, generalize, failure modes, authentic data, sources of difficulty, calibrating the difficulty, synthetic cases, overly adversarial

4.5 Assessing evaluation data quality

This section establishes the need to assess evaluation data quality before trusting results, regardless of how the data was obtained.

Key terms: evaluation data quality

4.5.1 Fitness to the evaluation task

This section explains how evaluation datasets can mismatch a task and how to check their fitness before relying on their results.

Key terms: user intent, operational constraints, good behavior, task definitions, annotation guidelines, success criteria, dataset samples, behavior sanity check, baseline, pilot evaluation

4.5.2 Accuracy and correctness

This section explains how inaccuracies and context mismatches in evaluation data can undermine results, and how to check for them.

Key terms: ground-truth facts, mislabeled examples, annotation errors, context-dependent, annotation guidelines, temporal validity, manual audit, representative sample, systematic disagreement, task definition

4.5.3 Coverage and representativeness

This section explains why evaluation datasets need relevant coverage and representativeness and how to assess both.

Key terms: coverage, representativeness, demographic skew, distributions, inputs, metadata, labels, external datasets, stress-test coverage, scenario checklists

4.5.4 Absence of annotation artifacts and shortcuts

This section explains how annotation artifacts can inflate evaluation scores and how to detect them using baselines and targeted inspection.

Key terms: annotation artifacts, spurious patterns, label leakage, trivial baselines, shallow baselines, surface correlations, high confidence, neutralize

4.6 References

This section lists references on evaluation datasets, benchmarks, and related research.

4.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 3: Picking evaluation metrics Chapter 5: Acquiring judgments →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.