liveBook · Manning

Question answering

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Question answering is a language-processing task in which a system generates or extracts answers to questions about provided content.

Question answering (QA) measures consistency by turning each target sentence into factual questions, then comparing answers extracted from multiple reference samples. section 5.2.3

Apply the method in two steps:

  • Generate questions about the target sentence, with each question aimed at a specific factual claim. section 5.2.3
  • Extract answers from every reference sample and compare them; matching answers indicate consistency, while variation indicates inconsistency. section 5.2.3

Historical date claim

For “the Pantheon was completed in 125 AD,” generate “When was the Pantheon completed?” Answers of “125 AD” across samples support consistency; differing answers indicate inconsistency. section 5.2.3

QA enables fine-grained verification, but depends on question-generation quality: weak questions can miss important claims or target irrelevant details. section 5.2.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains a two-step verification procedure and its limitation, but stops short of implementation details.
Examples
Uses a Pantheon completion-date claim and compares extracted answers across samples.
What it assumes
Assumes target sentences and reference samples, while explaining question generation and answer comparison from the procedure’s starting point.
What you can do after
Evaluate fine-grained factual consistency by generating claim-focused questions and comparing answers across samples.
How it connects
Connects question generation with consistency checking, factual claims, reference samples, and hallucination detection.

Also explained in

Compare how 6 books explain question answering →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.