liveBook · Manning

Reference-based evaluation

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A reference-based evaluation method compares an AI output with one or more authoritative human-created or human-verified reference answers or source materials.

Two AI responses can mean the same thing while using different words. Reference-based evaluation compares a candidate response with an answer or source material that defines correctness. The comparison may consider wording, meaning, or factual consistency rather than requiring an exact match. section 1.1 section 4.4.2

For open-ended tasks, human-created or human-verified answers form a golden dataset. Reference-based evaluation uses these answers as a basis for judgment; reference-free evaluation instead assesses outputs without gold-standard answers. A useful reference must fit the task and its intended context. section 1.1.2 section 1.5

A basic reference-based workflow:

  • Choose test inputs and obtain high-quality reference answers from experts. section 1.1.2
  • Run the AI system on the same inputs to produce candidate responses. section 1.1.2
  • Compare each candidate with its reference using an evaluator, and interpret the resulting scores in light of the task. section 1.1.2 section 1.5
Figure 1.4 Reference-based evaluation workflow. Human experts create reference answers from a test set. The AI system generates candidate responses from the same inputs. The evaluator then compares each candidate's response with the corresponding reference response, producing scores.

The workflow shows how expert answers and candidate responses are paired for scoring. section 1.1.2

Translation

For the English input “I am hungry,” a candidate French translation is “Je suis faim.” Comparing it with the human reference “J'ai faim” reveals that the candidate is grammatically incorrect. section 1.1.2

References can take different forms:

  • Human-written answers or summaries support comparisons of information overlap, factual consistency, and semantic equivalence. section 4.4.2
  • Retrieved documents or source texts can ground a check for unsupported claims in generated answers or summaries. section 4.4.2 section 5.3
  • Test suites and specifications can define correctness for generated code. section 4.4.2

The comparison method matters. Word-overlap metrics can miss equivalent responses with different wording, while semantic metrics and LLM judges aim to capture meaning. Human comparison can be costly, especially when subject-matter expertise is needed. No score is a universal measure of quality. section 1.1 section 1.1.2 section 1.5

In a related reference-guided grading technique, a judge first generates a solution in a separate context, then uses it to assess candidate answers. This independently generated reference is not necessarily a human-verified gold answer; its purpose is to reduce the risk that an incorrect candidate contaminates the judge's reasoning. section 4.3.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains the comparison workflow and reference types, including independently generated judge solutions, without implementing a specific scoring metric.
Examples
Illustrates comparison with a French translation, retrieved documents, reference summaries, and code test suites.
What you can do after
Evaluate and understand candidate responses against task-appropriate references while recognizing comparison and reference-quality limitations.
How it connects
Connects golden datasets, semantic similarity, LLM-as-a-judge, and faithfulness evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.