Ground truth is the reference standard used to judge whether a system’s outputs are correct, acceptable, or incorrect. Defining it early gives evaluators a shared basis for judgment and helps prevent fundamental disagreements. section 2.4.2
Two complementary definitions establish that reference point:
- An intensional definition sets criteria for correctness. For a medical chatbot, these might include medical accuracy, appropriate uncertainty, and safety warnings where needed. section 2.4.2
- An extensional definition supplies concrete examples, such as patient questions paired with acceptable and unacceptable responses. section 2.4.2
The initial standard need not cover every edge case, but it should resolve consequential ambiguities—for example, whether unsupported medical advice is acceptable or a conservative refusal counts as success. Criteria and examples can be refined later into detailed judgment rubrics and annotation guidelines. section 2.4.2

