liveBook · Manning

Reference-based metric

As explained in Evaluating AI Systems

A metric that evaluates system outputs by comparing them with one or more reference answers.

A reference-based metric evaluates a system output by comparing it with one or more answers treated as ground truth or acceptable exemplars. References can come from public benchmarks, labeled datasets, gold-standard outputs, or custom evaluation sets. section 3.2

Medical advice chatbot

For factual questions, compare a chatbot’s answers with clinician-verified references using exact match, semantic similarity, or a learned evaluation metric. section 3.2

Reference-based evaluation depends on having suitable answers to compare against. Where a correct answer does not exist or is too expensive to produce, reference-free metrics offer a contrast: they assess outputs without a fixed ground truth. section 3.2

How Evaluating AI Systems covers it

How deep it goes
Defines the metric category and contrasts it with reference-free evaluation without detailing implementations.
Examples
Illustrates evaluation with a medical advice chatbot and clinician-verified answers to factual questions.
What you can do after
Evaluate system answers against available references using comparison measures.
How it connects
Connects reference-based evaluation with exact match, semantic similarity, learned evaluation metrics, and reference-free metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.