Chapter 4: Acquiring evaluation data
This introduction explains what evaluation data can look like and why its suitability determines what an evaluation can reveal.
After this chapter you will be able to
- Forms of evaluation data. Describe what evaluation data is used for and how its form can vary across AI systems.
- Data limits on detection. Explain how the situations represented in evaluation data limit what an evaluation can detect, including how missing cases can create misleading results.
- Benchmarks and real usage. Distinguish the convenience of public benchmarks from the realism of pilot logs for evaluating a medical advice chatbot.
- Sampling and rare risks. Explain why proportionally sampling pilot logs may not adequately test rare but dangerous cases.
- Synthetic case trade-off. Describe the coverage opportunity and realism concern raised by synthetic patient cases.
- Evaluation data source trade-offs. Explain why the advantages, risks, and costs of different sources matter when building or selecting an evaluation dataset.
- Benchmark purpose and benefits. Explain what benchmark datasets contain and how their standardization supports evaluation and comparison.
- Benchmark suitability and validation. Explain when benchmark results may misrepresent deployment performance and why a benchmark should be checked for fitness to the intended evaluation.
- Annotation artifact shortcut. Trace how the NLI benchmark example revealed that models could exploit annotation artifacts instead of reasoning over sentence pairs.
- Operational data and context. Describe operational data and distinguish it from benchmarks, using an example from an AI application.
- Real-world evaluation strengths. Explain when operational data is especially useful for evaluation and why.
- Comparability and access limitations. Explain why internal data alone may be unsuitable for standardized comparisons or unavailable under access constraints.
- External real-world data sources. Describe external real-world data and distinguish it from benchmark datasets, using an example source.
- Breadth and deployment alignment. Explain when external real-world data is useful for evaluating generalization and why it may be less suitable for a tightly defined deployment context.
- Purpose-built human-authored data. Explain what human-authored evaluation data is and how it differs from naturally occurring data.
- Strengths and suitability trade-offs. Describe when human-authored data is useful and why it is less suitable for some evaluations.
- Synthetic evaluation data. Describe what synthetic evaluation data is and give an example of it.
- Useful conditions for synthesis. Explain when synthetic data is particularly useful for evaluation.
- Limits of synthetic evidence. Explain why synthetic data should not be the sole evidence for deployment, fairness, or safety decisions.
- Complementary data source strategy. Explain how different data sources can complement one another in a medical chatbot evaluation and how the evaluation’s purpose and constraints affect their selection.
- Evaluation data minimization. Explain data minimization using demographic attributes in a fairness evaluation as an example.
- Anonymization and pseudonymization. Distinguish anonymization from pseudonymization and explain why identifying details in free text make protection difficult.
- Random sampling trade-offs. Explain how random sampling supports overall performance estimation and what it can miss or inherit.
- Stratified subgroup sampling. Explain how stratified sampling supports subgroup analysis and identify its requirements and distribution trade-off.
- Adversarial sampling for risks. Explain why adversarial sampling is useful for stress testing but not for estimating overall performance.
- Temporal and contextual sampling. Explain how sampling across times or conditions helps detect changes in system performance.
- Evaluation goal and dataset size. Explain how the evaluation goal affects the amount of data needed, contrasting quick diagnostics with comparisons and risk-oriented evaluations.
- Task variability and coverage. Explain why variable, complex tasks and subgroup analyses often need more data, and why diversity of coverage matters more than raw volume alone.
- Metric stability diagnostic checks. Describe metric stability and how confidence intervals, resampling, or sensitivity analysis can reveal an insufficient or fragile evaluation set.
- Metric movability diagnostic checks. Explain how controlled contrasts and comparisons with other metrics can help distinguish an unresponsive metric from an inadequate dataset.
- Rule Based Data Generation. Explain how explicit patterns and constraints define synthetic evaluation examples and make them controlled and predictable.
- Template Instantiation And Labels. Describe how to build examples from scenario templates using structured lists, optional randomization, and deterministic logic for expected outputs.
- Controlled Testing Applications. Explain why rule-based data is useful for systematic stress tests, reproducible regression tests, and counterfactual or fairness probes.
- LLM synthesis and templates. Distinguish LLM-based data synthesis from rule-based generation and explain its advantage for evaluation data.
- Constrained generation prompts. Describe how to specify an initial prompt for LLM-generated evaluation data, using the patient-question example.
- Examples and topic guidance. Explain how examples, topic lists, and external materials can strengthen a synthesis prompt.
- Iterative generation and review. Trace the iterative process of generating, reviewing, and refining synthetic examples.
- Synthetic data trade-off. Explain how the consequences of false alarms and missed weaknesses influence whether to prioritize variety or correctness and plausibility in synthetic evaluation data.
- Synthetic and real data characteristics. Explain how comparing synthetic inputs with real examples can reveal a lack of representativeness.
- Alignment with real failure modes. Describe how real system failures can be used to check whether synthetic examples capture realistic difficulties.
- Synthetic case difficulty calibration. Explain how to use model performance on real data to assess the difficulty of synthetic cases.
- Evaluation data quality assessment. Explain why evaluation data should be critically assessed before its results are trusted, regardless of its source or generation method.
- Sources of task mismatch. Explain how differences in user intent, operational constraints, and assumptions about good behavior can make an apparently similar dataset unsuitable for an evaluation task.
- Definitions and sample inspection. Describe how to compare a dataset's documented task definition and a diverse sample of its items with the intended evaluation task.
- Baseline and pilot checks. Describe how a behavior sanity check with a baseline and a pilot with qualitative error analysis can reveal a dataset's mismatched notion of success.
- Errors and evaluation validity. Explain how factual and labeling errors in an evaluation dataset can distort assessment of a system.
- Context and temporal validity. Explain how context-dependent judgments, underspecified guidelines, and changing standards can make dataset labels inappropriate for an evaluation.
- Representative manual data audit. Describe how to audit a targeted, representative sample and interpret reviewer disagreement.
- Coverage and representativeness distinction. Distinguish coverage from representativeness and explain why both matter for evaluation conclusions.
- Relative representativeness and skew. Explain why representativeness depends on the intended setting and describe how collection choices or changing usage can create skew.
- Distribution and external comparisons. Describe how to assess representativeness by comparing dataset distributions with expected deployment patterns and relevant external sources.
- Scenario checklist stress testing. Describe how to stress-test coverage using a checklist of important scenarios, edge cases, and risk conditions.
- Annotation artifacts and shortcuts. Explain how spurious patterns in evaluation data can inflate scores without demonstrating the intended capability, and describe where such patterns can arise.
- Shallow baseline artifact detection. Explain how trivial or shallow baselines can flag a dataset that rewards surface correlations instead of the intended reasoning or semantic understanding.
- High confidence case inspection. Describe how to inspect highly confident correct cases for incidental patterns and test whether those patterns inflated evaluation results.
What it builds on
- What is a metric (chapter 3) · The role of metrics in measuring system performance.
- Defining the task to be evaluated (chapter 2) · The task being evaluated helps determine which situations are relevant.
- Choosing evaluation dimensions (chapter 2) · The dimensions chosen for evaluation shape which behaviors and risks matter.
- Defining evaluation goals (chapter 2) · The evaluation goal sets the purpose for which data is needed.
- Reliability (chapter 3) · Metric reliability concerns repeatability under identical conditions.
- Movability (chapter 3) · Metric movability is responsiveness to genuine changes in system quality.
- Defining an AI task well (chapter 2) · The evaluation task definition provides the basis for judging dataset fit.
Section by section
Introduction
This introduction explains what evaluation data can look like and why its suitability determines what an evaluation can reveal.
- Forms of evaluation data: Describe what evaluation data is used for and how its form can vary across AI systems.
Why it matters: Recognizing the forms evaluation data can take helps you identify what an evaluation needs to exercise and measure. - Data limits on detection: Explain how the situations represented in evaluation data limit what an evaluation can detect, including how missing cases can create misleading results.
Why it matters: Even a good metric can give a misleading picture if the evaluation data never exposes important failures.
Key terms: evaluation data, inputs, examples, test cases, metrics, subgroups, dangerous edge cases, behaviors, risks, capabilities
4.1 “What do we test it on?”
The section uses a medical chatbot scenario to illustrate trade-offs in choosing evaluation data, sampling cases, and using synthetic examples.
- Benchmarks and real usage: Distinguish the convenience of public benchmarks from the realism of pilot logs for evaluating a medical advice chatbot.
Why it matters: The data you choose can determine whether the evaluation tests the problem the chatbot will actually face. - Sensitive log reuse: Explain why using patient pilot logs requires more than simply anonymizing them.
Why it matters: Realistic evaluation data can carry obligations that limit how it may be reused. - Sampling and rare risks: Explain why proportionally sampling pilot logs may not adequately test rare but dangerous cases.
Why it matters: A sample that reflects typical usage may miss the dangerous situations an evaluation most needs to test. - Synthetic case trade-off: Describe the coverage opportunity and realism concern raised by synthetic patient cases.
Why it matters: Designed cases can expand coverage, but their usefulness depends on whether they are credible for the decision at hand.
Key terms: public datasets, benchmarks, logs, real usage, sensitive information, anonymize, legal basis, purpose limitation, patient privacy, sampling
4.2 Where to look for evaluation data
The section introduces evaluation data sources and the trade-offs involved in choosing among them.
- Evaluation data source trade-offs: Explain why the advantages, risks, and costs of different sources matter when building or selecting an evaluation dataset.
Why it matters: Considering source trade-offs helps you make a more deliberate choice of evaluation data.
Key terms: evaluation data sources, advantages, risks, costs
4.2.1 Benchmarks
This section explains what benchmark datasets offer, when they are useful or misleading, and how annotation artifacts can inflate apparent performance.
- Benchmark purpose and benefits: Explain what benchmark datasets contain and how their standardization supports evaluation and comparison.
Why it matters: Shared test data can make comparisons faster and more consistent. - Benchmark discovery and access: Describe where evaluators can look for benchmarks and why access may vary.
Why it matters: Knowing where to look and that access may be restricted helps you find a usable benchmark. - Benchmark suitability and validation: Explain when benchmark results may misrepresent deployment performance and why a benchmark should be checked for fitness to the intended evaluation.
Why it matters: A strong benchmark score does not necessarily predict performance for your users or risks. - Annotation artifact shortcut: Trace how the NLI benchmark example revealed that models could exploit annotation artifacts instead of reasoning over sentence pairs.
Why it matters: A benchmark can reward shortcuts that do not demonstrate the ability it is supposed to test.
Key terms: benchmark datasets, reference outputs, reproducible evaluations, baseline analysis, academic repositories, community leaderboards, licensing, representativeness, freshness, difficulty
4.2.2 Internal or operational data
This section describes internal operational data, where to find it, and when its realism or limitations make it suitable for evaluation.
- Operational data and context: Describe operational data and distinguish it from benchmarks, using an example from an AI application.
Why it matters: Recognizing operational data helps you find examples that reflect how a system is actually used. - Organizational data access: Describe where operational data can be found and how access and preparation can vary.
Why it matters: Knowing where records live and who may need to help makes operational data acquisition more practical. - Real-world evaluation strengths: Explain when operational data is especially useful for evaluation and why.
Why it matters: The right real-world examples can reveal whether a system works in its actual operating context. - Comparability and access limitations: Explain why internal data alone may be unsuitable for standardized comparisons or unavailable under access constraints.
Why it matters: These limitations help you decide when to supplement internal data or choose another source.
Key terms: internal data, operational data, benchmarks, production logs, organizational databases, internal monitoring pipelines, observability tools, data warehouses, practical effectiveness, user alignment
4.2.3 Real-world external data
This section explains what external real-world evaluation data is and when its breadth is useful or its lack of deployment-specific alignment is limiting.
- External real-world data sources: Describe external real-world data and distinguish it from benchmark datasets, using an example source.
Why it matters: Recognizing this source helps you find evaluation material beyond your organization and understand how it differs from a benchmark. - Breadth and deployment alignment: Explain when external real-world data is useful for evaluating generalization and why it may be less suitable for a tightly defined deployment context.
Why it matters: This trade-off helps you choose data that matches the question your evaluation needs to answer.
Key terms: external real-world datasets, benchmark datasets, generalization, stress-testing, domain-shift evaluation, fairness assessments, robustness checks, deployment context
4.2.4 Human-authored data
This section explains human-authored evaluation data and when its control and flexibility outweigh its limitations.
- Purpose-built human-authored data: Explain what human-authored evaluation data is and how it differs from naturally occurring data.
Why it matters: Purpose-built examples let you tailor an evaluation to the system and questions you care about. - Strengths and suitability trade-offs: Describe when human-authored data is useful and why it is less suitable for some evaluations.
Why it matters: Knowing these trade-offs helps you choose human-authored data for targeted tests without mistaking it for a scalable picture of natural usage.
Key terms: human-authored data, inputs, labels, judgments, controlled conditions, failure modes, edge cases, representativeness, production monitoring, drift detection
4.2.5 Synthetic data
This section defines synthetic evaluation data and explains when it is useful and why it cannot replace real-world evidence.
- Synthetic evaluation data: Describe what synthetic evaluation data is and give an example of it.
Why it matters: Recognizing synthetic data helps you distinguish generated test cases from evidence collected from real interactions. - Useful conditions for synthesis: Explain when synthetic data is particularly useful for evaluation.
Why it matters: These conditions help you decide when generated cases can provide useful evaluation evidence. - Limits of synthetic evidence: Explain why synthetic data should not be the sole evidence for deployment, fairness, or safety decisions.
Why it matters: Generated tests alone can make a system look more trustworthy than its behavior in the real world warrants.
Key terms: synthetic evaluation data, templated prompts, simulated environments, synthetic data, stress testing, operational data, realistic user behavior, natural settings, ambiguity, diversity
4.2.6 Choosing the right source of evaluation data
This section explains how to combine evaluation data sources and acquire data responsibly under practical, legal, and ethical constraints.
- Complementary data source strategy: Explain how different data sources can complement one another in a medical chatbot evaluation and how the evaluation’s purpose and constraints affect their selection.
Why it matters: A deliberate mix can cover risks and realistic behavior that no single dataset captures. - Legal compliance requirements: Describe how data-protection requirements apply to evaluation data, including the possibility that internal testing still needs formal oversight or agreements.
Why it matters: Evaluation data handling can carry legal obligations even when the work is internal. - Evaluation data minimization: Explain data minimization using demographic attributes in a fairness evaluation as an example.
Why it matters: Limiting collection to what the evaluation needs reduces unnecessary exposure of sensitive information. - Anonymization and pseudonymization: Distinguish anonymization from pseudonymization and explain why identifying details in free text make protection difficult.
Why it matters: Knowing the difference helps protect people while retaining data useful for evaluation. - Secure data handling: Describe safeguards for storing and processing sensitive evaluation examples and judgments.
Why it matters: Evaluation datasets can persist and spread across teams unless access and retention are governed. - Participant consent and welfare: Describe what people contributing to evaluation datasets should be told about their participation.
Why it matters: People contributing evaluation data need enough information to understand their participation.
Key terms: benchmarks, internal operational data, synthetic data, human-generated data, legal compliance, GDPR, HIPAA, data-processing agreements, privacy impact assessments, data minimization
4.3 Collecting and sampling evaluation data
This section introduces the need to make deliberate choices when collecting or sampling evaluation data.
4.3.1 Sampling methods and strategies
This section compares sampling strategies for estimating performance, analyzing subgroups, finding weaknesses, and detecting changes across conditions.
- Random sampling trade-offs: Explain how random sampling supports overall performance estimation and what it can miss or inherit.
Why it matters: A random sample can support an overall estimate without guaranteeing coverage of rare risks or freedom from bias. - Stratified subgroup sampling: Explain how stratified sampling supports subgroup analysis and identify its requirements and distribution trade-off.
Why it matters: Subgroup coverage can reveal differences that an overall sample may obscure, but the resulting mix may not mirror reality. - Adversarial sampling for risks: Explain why adversarial sampling is useful for stress testing but not for estimating overall performance.
Why it matters: A deliberately difficult test set can expose risks that an overall performance estimate may hide. - Observed failure error analysis: Describe how error analysis turns observed failures into evaluation cases and why it complements other sampling strategies.
Why it matters: Real failures can become useful diagnostic and regression cases without being mistaken for a representative test set. - Temporal and contextual sampling: Explain how sampling across times or conditions helps detect changes in system performance.
Why it matters: Performance observed in one period or setting may not hold when conditions change.
Key terms: random sampling, overall system performance, stratified sampling, strata, subgroup performance, adversarial sampling, stress testing, risk discovery, error analysis, observed failures
4.3.2 Determining how much evaluation data you need
This section explains how evaluation goals, task variability, metric stability, and metric movability help determine whether an evaluation dataset is sufficient.
- Evaluation goal and dataset size: Explain how the evaluation goal affects the amount of data needed, contrasting quick diagnostics with comparisons and risk-oriented evaluations.
Why it matters: The right dataset size depends on what your evaluation needs to support. - Task variability and coverage: Explain why variable, complex tasks and subgroup analyses often need more data, and why diversity of coverage matters more than raw volume alone.
Why it matters: More examples help only if they cover the conditions your evaluation needs to examine. - Metric stability diagnostic checks: Describe metric stability and how confidence intervals, resampling, or sensitivity analysis can reveal an insufficient or fragile evaluation set.
Why it matters: Stability checks reveal when conclusions depend too heavily on the particular examples sampled. - Incremental dataset growth: Describe how adding examples in batches and tracking a reliable metric can signal when an evaluation set is large enough.
Why it matters: A convergence curve offers a practical signal for when further data collection may add little. - Metric movability diagnostic checks: Explain how controlled contrasts and comparisons with other metrics can help distinguish an unresponsive metric from an inadequate dataset.
Why it matters: A flat score can reflect a weak metric rather than a dataset that is already sufficient.
Key terms: evaluation goal, quick diagnostics, benchmark-style comparisons, risk assessment, task variability, task complexity, coverage, subgroups, metric stability, confidence intervals
4.4 Synthesizing evaluation data
This section introduces rule-based and LLM-based approaches to generating synthetic evaluation data.
4.4.1 Rule-based or template-based generation
This section explains how rule-based templates generate controlled synthetic evaluation data and where that control is useful in testing.
- Rule Based Data Generation: Explain how explicit patterns and constraints define synthetic evaluation examples and make them controlled and predictable.
Why it matters: Explicit rules make it easier to understand and check what each evaluation example tests. - Template Instantiation And Labels: Describe how to build examples from scenario templates using structured lists, optional randomization, and deterministic logic for expected outputs.
Why it matters: This process lets you generate varied examples while retaining explicit control over their construction and expected results. - Controlled Testing Applications: Explain why rule-based data is useful for systematic stress tests, reproducible regression tests, and counterfactual or fairness probes.
Why it matters: Controlled and repeatable examples help isolate what changed and test behavior under specific conditions.
Key terms: rule-based generation, synthetic evaluation data, templates, logical constraints, placeholders, controlled vocabularies, structured domain lists, randomization, deterministic logic, expected outputs
4.4.2 LLM-based data synthesis
This section explains how to prompt an LLM to generate diverse evaluation examples and iteratively review and improve them.
- LLM synthesis and templates: Distinguish LLM-based data synthesis from rule-based generation and explain its advantage for evaluation data.
Why it matters: This distinction helps you choose how to generate the kinds of test inputs you need. - Constrained generation prompts: Describe how to specify an initial prompt for LLM-generated evaluation data, using the patient-question example.
Why it matters: A concrete prompt directs the model toward examples relevant to the evaluation. - Examples and topic guidance: Explain how examples, topic lists, and external materials can strengthen a synthesis prompt.
Why it matters: These additions help steer both the form and the coverage of generated test cases. - Iterative generation and review: Trace the iterative process of generating, reviewing, and refining synthetic examples.
Why it matters: Review and revision help turn an initial batch into more useful evaluation data.
Key terms: LLM-based data synthesis, synthetic examples, rule-based generation, base prompt, constraints, realistic patient questions, examples, topic lists, external materials, stylistic and difficulty controls
4.4.3 Handling the variety vs. correctness/plausibility trade-off
This section explains how the consequences of evaluation mistakes guide the balance between variety and correctness or plausibility in synthetic data.
- Synthetic data trade-off: Explain how the consequences of false alarms and missed weaknesses influence whether to prioritize variety or correctness and plausibility in synthetic evaluation data.
Why it matters: The right balance depends on which evaluation mistakes would do more harm.
Key terms: variety, correctness, plausibility, false alarm
4.4.4 Validating synthetic data against real data
This section explains how to validate synthetic evaluation data against real data by comparing characteristics, failure modes, and difficulty.
- Synthetic and real data characteristics: Explain how comparing synthetic inputs with real examples can reveal a lack of representativeness.
Why it matters: This comparison helps reveal whether synthetic tests resemble the situations the system will encounter. - Alignment with real failure modes: Describe how real system failures can be used to check whether synthetic examples capture realistic difficulties.
Why it matters: Synthetic examples are more useful when they exercise difficulties the system actually encounters. - Synthetic case difficulty calibration: Explain how to use model performance on real data to assess the difficulty of synthetic cases.
Why it matters: Performance differences can warn that synthetic tests are unrealistically easy or hard.
Key terms: distributions, characteristics, topical coverage, generalize, failure modes, authentic data, sources of difficulty, calibrating the difficulty, synthetic cases, overly adversarial
4.5 Assessing evaluation data quality
This section establishes the need to assess evaluation data quality before trusting results, regardless of how the data was obtained.
- Evaluation data quality assessment: Explain why evaluation data should be critically assessed before its results are trusted, regardless of its source or generation method.
Why it matters: Results are only worth trusting after you have examined the quality of the data behind them.
Key terms: evaluation data quality
4.5.1 Fitness to the evaluation task
This section explains how evaluation datasets can mismatch a task and how to check their fitness before relying on their results.
- Sources of task mismatch: Explain how differences in user intent, operational constraints, and assumptions about good behavior can make an apparently similar dataset unsuitable for an evaluation task.
Why it matters: A high score can mislead you if the dataset rewards behavior that is wrong for your application. - Definitions and sample inspection: Describe how to compare a dataset's documented task definition and a diverse sample of its items with the intended evaluation task.
Why it matters: Comparing both descriptions and actual examples can reveal mismatches that a dataset's name or documentation alone might hide. - Baseline and pilot checks: Describe how a behavior sanity check with a baseline and a pilot with qualitative error analysis can reveal a dataset's mismatched notion of success.
Why it matters: These checks reveal when dataset scores disagree with the behavior your application actually needs.
Key terms: user intent, operational constraints, good behavior, task definitions, annotation guidelines, success criteria, dataset samples, behavior sanity check, baseline, pilot evaluation
4.5.2 Accuracy and correctness
This section explains how inaccuracies and context mismatches in evaluation data can undermine results, and how to check for them.
- Errors and evaluation validity: Explain how factual and labeling errors in an evaluation dataset can distort assessment of a system.
Why it matters: A dataset that fits the task can still give misleading results if its answers or labels are wrong. - Context and temporal validity: Explain how context-dependent judgments, underspecified guidelines, and changing standards can make dataset labels inappropriate for an evaluation.
Why it matters: A label can seem reasonable in its original setting yet be misleading or unsafe in yours. - Representative manual data audit: Describe how to audit a targeted, representative sample and interpret reviewer disagreement.
Why it matters: A focused audit can expose problems that a dataset's apparent relevance does not reveal. - Documentation and external verification: Describe how dataset documentation and current authoritative sources can reveal accuracy risks.
Why it matters: Provenance and independent references provide complementary ways to spot questionable data.
Key terms: ground-truth facts, mislabeled examples, annotation errors, context-dependent, annotation guidelines, temporal validity, manual audit, representative sample, systematic disagreement, task definition
4.5.3 Coverage and representativeness
This section explains why evaluation datasets need relevant coverage and representativeness and how to assess both.
- Coverage and representativeness distinction: Distinguish coverage from representativeness and explain why both matter for evaluation conclusions.
Why it matters: A dataset can yield confident-looking results while missing important behaviors or misrepresenting real use. - Relative representativeness and skew: Explain why representativeness depends on the intended setting and describe how collection choices or changing usage can create skew.
Why it matters: A dataset that accurately reflects where it came from can still be a poor fit for the system being evaluated. - Distribution and external comparisons: Describe how to assess representativeness by comparing dataset distributions with expected deployment patterns and relevant external sources.
Why it matters: Distribution comparisons can expose gaps that are not obvious from dataset size or individual examples. - Scenario checklist stress testing: Describe how to stress-test coverage using a checklist of important scenarios, edge cases, and risk conditions.
Why it matters: A checklist helps reveal important risks that a dataset cannot meaningfully test.
Key terms: coverage, representativeness, demographic skew, distributions, inputs, metadata, labels, external datasets, stress-test coverage, scenario checklists
4.5.4 Absence of annotation artifacts and shortcuts
This section explains how annotation artifacts can inflate evaluation scores and how to detect them using baselines and targeted inspection.
- Annotation artifacts and shortcuts: Explain how spurious patterns in evaluation data can inflate scores without demonstrating the intended capability, and describe where such patterns can arise.
Why it matters: Recognizing shortcuts helps you avoid mistaking a high evaluation score for the capability you intended to measure. - Shallow baseline artifact detection: Explain how trivial or shallow baselines can flag a dataset that rewards surface correlations instead of the intended reasoning or semantic understanding.
Why it matters: A simple baseline can reveal that an evaluation is easier to game than its intended task suggests. - High confidence case inspection: Describe how to inspect highly confident correct cases for incidental patterns and test whether those patterns inflated evaluation results.
Why it matters: This diagnostic connects a suspected shortcut to its effect on the evaluation score.
Key terms: annotation artifacts, spurious patterns, label leakage, trivial baselines, shallow baselines, surface correlations, high confidence, neutralize
4.6 References
This section lists references on evaluation datasets, benchmarks, and related research.
4.7 Summary
Where this is used later
- Deciding who will judge (chapter 5) · People recruited to judge outputs are among those whose consent and welfare should be considered.
- Writing instructions that minimize ambiguity and vagueness (chapter 5) · Clear judgment instructions help avoid the underspecified criteria described here.
Concepts explained in this chapter
- synthetic dataArtificially generated data designed to represent selected characteristics or scenarios rather than collected directly from real-world events.
- evaluation datasetData on which an AI system is assessed and from which evaluation measures are computed.
- diversityThe degree of diversity among examples in a dataset.
- annotation artifactAn unintended pattern in evaluation data, arising from its creation or annotation, that can let a system score well without demonstrating the intended capability.
- metric stabilityThe degree to which a metric's value remains similar when the evaluation dataset changes slightly.
- sample sizeSample size is the number of observations or examples included in a study or evaluation.
- benchmark datasetA dataset used as a common reference for evaluating system performance.
- spurious correlationA correlation between data features and labels that does not reflect the underlying relationship a system is meant to learn.
- evaluation taskAn evaluation task is an activity an AI system is expected to perform and whose outcomes can be assessed.
- task definitionA task definition specifies the activity an AI system is expected to perform and the conditions that define its scope and success.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
