Two AI responses can mean the same thing while using different words. Reference-based evaluation compares a candidate response with an answer or source material that defines correctness. The comparison may consider wording, meaning, or factual consistency rather than requiring an exact match. section 1.1 section 4.4.2
For open-ended tasks, human-created or human-verified answers form a golden dataset. Reference-based evaluation uses these answers as a basis for judgment; reference-free evaluation instead assesses outputs without gold-standard answers. A useful reference must fit the task and its intended context. section 1.1.2 section 1.5
A basic reference-based workflow:
- Choose test inputs and obtain high-quality reference answers from experts. section 1.1.2
- Run the AI system on the same inputs to produce candidate responses. section 1.1.2
- Compare each candidate with its reference using an evaluator, and interpret the resulting scores in light of the task. section 1.1.2 section 1.5
Figure 1.4 Reference-based evaluation workflow. Human experts create reference answers from a test set. The AI system generates candidate responses from the same inputs. The evaluator then compares each candidate's response with the corresponding reference response, producing scores.
The workflow shows how expert answers and candidate responses are paired for scoring. section 1.1.2
Translation
For the English input “I am hungry,” a candidate French translation is “Je suis faim.” Comparing it with the human reference “J'ai faim” reveals that the candidate is grammatically incorrect. section 1.1.2
References can take different forms:
- Human-written answers or summaries support comparisons of information overlap, factual consistency, and semantic equivalence. section 4.4.2
- Retrieved documents or source texts can ground a check for unsupported claims in generated answers or summaries. section 4.4.2 section 5.3
- Test suites and specifications can define correctness for generated code. section 4.4.2
The comparison method matters. Word-overlap metrics can miss equivalent responses with different wording, while semantic metrics and LLM judges aim to capture meaning. Human comparison can be costly, especially when subject-matter expertise is needed. No score is a universal measure of quality. section 1.1 section 1.1.2 section 1.5
In a related reference-guided grading technique, a judge first generates a solution in a separate context, then uses it to assess candidate answers. This independently generated reference is not necessarily a human-verified gold answer; its purpose is to reduce the risk that an incorrect candidate contaminates the judge's reasoning. section 4.3.4
