An evaluation dataset provides examples and judgments for assessing a model or decision procedure. Depending on the task, it can contain expert-written reference answers, human labels, or query-document relevance judgments. section 1.1.2 section 6.2.4
For hallucination detection, build a validation set of 100–200 human-annotated responses, run the detector, and use its precision-recall curve to choose a threshold. Recalibrate periodically as models, users, or inputs change; the preferred threshold depends on the cost of false positives versus false negatives. section 5.4.3
For retrieval evaluation, assemble:
- A corpus of documents. section 6.2.4
- Queries, drawn from search logs or generated synthetically. section 6.2.4
- Relevance judgments mapping queries to relevant documents. Human experts provide authoritative labels at higher cost; LLM judges and synthetic queries can scale labeling, but may introduce bias or mismatch real user queries. section 6.2.4
Figure 1.4 Reference-based evaluation workflow. Human experts create reference answers from a test set. The AI system generates candidate responses from the same inputs. The evaluator then compares each candidate's response with the corresponding reference response, producing scores.
Reference-based evaluation compares a system’s candidate response with a human-created reference for the same input. section 1.1.2








































