liveBook · Manning

Closed-domain task

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A closed-domain setting is an application context in which authoritative reference material is available for validating generated outputs.

A closed-domain task uses a specific context, document, or database as the boundary for what an answer may claim. The supplied material acts as a source of truth, making outputs testable against verifiable evidence rather than subjective truth. section 5.1 section 5.1.1

Classify the task before selecting a hallucination detector:

  • Identify a supplied source, such as retrieved documents, a source text, or database records. section 5.1.1 section 5.3
  • Compare the generated output with that source to test contextual grounding: whether the system stayed within the provided material. section 5.1.1
  • Verify individual claims with natural language inference, which classifies the source relationship as entailment, contradiction, or neutral. section 5.1.1

Reference-based verification

A retrieval-augmented generation system can compare its answer with retrieved documents; a summarizer can compare its summary with the source text; structured question answering can check claims against database records. These reference-based signals are stronger when grounding material is available. section 5.3 section 5.1.1

Figure 5.1 A flow chart for choosing a hallucination detection strategy. Closed-domain tasks can use grounding based on sources to verify claims. Open-domain tasks require self-consistency checks, or rely on external fact retrieval, or other factual alignment signals.

Use the decision flow to connect closed-domain tasks with source grounding and distinguish them from open-domain tasks that require other factual-alignment signals. section 5.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains source boundaries, contextual grounding, and NLI verification, but defers detailed detection methods to later treatment.
Examples
Uses retrieval-augmented generation, summarization, deep research, document question answering, and structured question answering as examples.
What it assumes
Assumes generated outputs and reference material, while defining contextual grounding and NLI for beginners.
What you can do after
Evaluate generated claims against source material and apply NLI to detect entailment, contradiction, or neutrality.
How it connects
Connects contextual grounding with hallucination detection, factual alignment, natural language inference, retrieval-augmented generation, and summarization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.