liveBook · Manning

Source of truth

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A source of truth is the authoritative context or information against which an output is expected to conform.

A source of truth is the authoritative context or information an output must conform to. In model alignment, a written specification serves this role: its clauses define expected behavior for both training and evaluation. section 8.4.4

Make the source of truth operational by:

  • Translating each specification clause into a test prompt with a known expected behavior. section 8.4.4
  • Running the resulting evaluation suite whenever the model, specification, or deployment changes. section 8.4.4 section 8.5.4
  • Versioning the specification and tying every deployed model to the version used for training. section 8.5.4
  • Turning every production failure into either additional training data for an existing clause or a new clause. section 8.5.4

Financial research assistant

Require the assistant to cite the source paragraph for every numerical claim and say “I don’t know” when the answer is absent from retrieved context. Test each rule automatically, then block deployment when the suite fails. section 8.4.4

The same pattern applies to closed-domain generation: the provided context is the source of truth, and faithfulness checks determine whether an answer stays grounded before other quality dimensions are scored. section 5.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains clauses, executable tests, versioning, failure handling, and deployment gates, while stopping short of specifying training algorithms.
Examples
Uses citation checking, unknown-answer prompts, specification versions, production incidents, and deploy-gate test suites.
What it assumes
Assumes basic model evaluation and production deployment concepts while defining the alignment role of an explicit specification.
What you can do after
Operate/configure an alignment process that converts requirements into tests, traces failures, and blocks nonconforming deployments.
How it connects
Connects specifications with evaluation suites, regression testing, model alignment, contextual grounding, and hallucination detection.

Also explained in

Compare how 13 books explain source of truth →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.