chapter four
4 Acquiring evaluation data
This chapter covers
- Identifying and selecting sources of AI evaluation data.
- Collecting and sampling evaluation data for different evaluation goals and contexts.
- Creating and validating synthetic evaluation data
- Assessing the quality of evaluation data
“The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.”
— John Tukey
Every evaluation needs data, namely inputs, examples, and test cases that will be fed to the system and used to compute the metrics you have chosen. What counts as evaluation data varies widely across domains and system types. In a medical image classification system, for example, evaluation data may consist of labeled X-ray or MRI images annotated for the presence of tumors or other conditions. For a RAG-based question answering system, it may include user questions, retrieved documents, and reference answers used to assess factual grounding and correctness. And for agentic systems, it may take the form of simulated or real-world environments designed to test them under a range of conditions.