liveBook · Manning

Label

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A gold standard is an expert-produced reference answer or output treated as the target for evaluation.

A gold standard is an expert-produced reference answer or output treated as the target for evaluating an AI-generated candidate. Reference-based evaluation compares the candidate with one or more such outputs, while reference-free evaluation judges quality without them. section 1.1 section 2.1.1

Use gold standards when:

  • A task has a trusted expert answer, such as a translation, summary, or other target output. section 1.1
  • Several valid answers exist, making exact matching insufficient; multiple references can represent acceptable variation. section 2.1.1
  • You need automated signals such as word overlap or semantic similarity, while recognizing that these remain proxies for quality. section 2.1.1 section 7.1.1

Evaluating a translation

Given a Chinese sentence, compare two machine translations with three professional human translations. The better candidate shares more appropriate words and phrases with the references, though overlap alone does not fully measure meaning or quality. section 2.1.1

Figure 1.1 A taxonomy of AI evaluation methods, distinguishing between verifiable tasks and open-ended tasks.

Locate gold standards on the reference-based branch for open-ended tasks; verifiable tasks instead use objectively checkable answers. section 1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines references and contrasts them with candidates, but stops short of detailing how experts create or validate reference sets.
Examples
Uses professional translation references to compare machine-generated candidates and motivate BLEU-style overlap measurement.
What it assumes
Assumes familiarity with generated outputs and evaluation, while introducing candidates, references, and the verifiable/open-ended task distinction.
What you can do after
Evaluate/understand outputs by comparing candidates with expert references while interpreting metric scores as limited proxies.
How it connects
Connects reference-based evaluation with open-ended tasks, lexical-overlap metrics, semantic similarity, and human judgment.

Also explained in

Compare how 29 books explain label →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.