chapter four

4 Acquiring evaluation data

 

This chapter covers

  • Identifying and selecting sources of AI evaluation data.
  • Collecting and sampling evaluation data for different evaluation goals and contexts.
  • Creating and validating synthetic evaluation data
  • Assessing the quality of evaluation data
“The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.”

— John Tukey

Every evaluation needs data, namely inputs, examples, and test cases that will be fed to the system and used to compute the metrics you have chosen. What counts as evaluation data varies widely across domains and system types. In a medical image classification system, for example, evaluation data may consist of labeled X-ray or MRI images annotated for the presence of tumors or other conditions. For a RAG-based question answering system, it may include user questions, retrieved documents, and reference answers used to assess factual grounding and correctness. And for agentic systems, it may take the form of simulated or real-world environments designed to test them under a range of conditions.

4.1 “What do we test it on?”

4.2 Where to look for evaluation data

4.2.1 Benchmarks

4.2.2 Internal or operational data

4.2.3 Real-world external data

4.2.4 Human-authored data

4.2.5 Synthetic data

4.2.6 Choosing the right source of evaluation data

4.3 Collecting and sampling evaluation data

4.3.1 Sampling methods and strategies

4.3.2 Determining how much evaluation data you need

4.4 Synthesizing evaluation data

4.4.1 Rule-based or template-based generation

4.4.2 LLM-based data synthesis

4.4.3 Handling the variety vs. correctness/plausibility trade-off

4.4.4 Validating synthetic data against real data

4.5 Assessing evaluation data quality

4.5.1 Fitness to the evaluation task

4.5.2 Accuracy and correctness

4.5.3 Coverage and representativeness

4.5.4 Absence of annotation artifacts and shortcuts

4.6 References

4.7 Summary