liveBook · Manning

Ground truth

As explained in Evaluating AI Systems

Ground truth is the reference standard used to judge whether system outputs are correct, acceptable, or incorrect.

Ground truth is the reference standard used to judge whether a system’s outputs are correct, acceptable, or incorrect. Defining it early gives evaluators a shared basis for judgment and helps prevent fundamental disagreements. section 2.4.2

Two complementary definitions establish that reference point:

  • An intensional definition sets criteria for correctness. For a medical chatbot, these might include medical accuracy, appropriate uncertainty, and safety warnings where needed. section 2.4.2
  • An extensional definition supplies concrete examples, such as patient questions paired with acceptable and unacceptable responses. section 2.4.2

The initial standard need not cover every edge case, but it should resolve consequential ambiguities—for example, whether unsupported medical advice is acceptable or a conservative refusal counts as success. Criteria and examples can be refined later into detailed judgment rubrics and annotation guidelines. section 2.4.2

How Evaluating AI Systems covers it

How deep it goes
Distinguishes criteria-based and example-based definitions while deferring detailed judgment rubrics and annotation guidelines.
Examples
Illustrates the standard with medical chatbot criteria and patient questions paired with acceptable and unacceptable responses.
What you can do after
Evaluate and understand whether a task’s initial ground truth resolves fundamental disagreements without specifying every edge case.
How it connects
Links ground truth to judgment rubrics and annotation guidelines.

Also explained in

Compare how 2 books explain ground truth →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.