liveBook · Manning

Fact-single question

As explained in Retrieval Augmented Generation, The Foundational Ideas

A fact_single question is an evaluation question whose answer can be obtained as a single factual item from context.

A fact-single question asks for one factual item stated directly in the supplied context. For example, “What supply voltage should I use?” can be answered by locating a single matching statement rather than combining evidence or inferring a conclusion. section 14.4

Fact-single questions are useful for testing direct retrieval, but they can make a RAG system appear stronger than it is. Public QA benchmarks commonly contain 60–80% of them, while enterprise users often ask summary and reasoning questions. The best hybrid-search text weight can also change by question type, so a fact-single-heavy test set may tune retrieval for the wrong workload. section 14.4 section 14.7

Use fact-single questions to:

  • Check whether retrieval finds the passage containing an explicit answer. section 14.4
  • Generate grounded evaluation pairs by extracting factual statements first, then reversing each statement into a question. section 14.4.1
  • Label results separately from summary, reasoning, and unanswerable questions so evaluation scores reveal query-type weaknesses. section 14.4 section 14.4.1

Grounded generation pattern

Given a corpus passage, extract an individual statement such as a specification or requirement. Ask an LLM to write a specific question that the statement answers, and store the passage, question, statement as the reference answer, and the label “fact_single.” This keeps the answer grounded in actual corpus content and reduces hallucination in the evaluation data. section 14.4.1

Table 14.3 Question-context taxonomy for RAG evaluation datasets
Class
Description
Example query
Retrieval difficulty
fact_single
Answer is one unit of information, directly stated in context
"What supply voltage should I use?"
Easiest: direct lexical/semantic match
summary
Answer requires multiple pieces from context
"Summarize the key findings."
Medium: must retrieve the broad context
reasoning
Answer must be inferred, not directly stated
"Has consumption increased over 5 years?"
Hardest: context doesn't contain the answer verbatim
unanswerable
Answer cannot be derived from the context
"Is a tomato a fruit or a vegetable?" (asked of a coffee machine patent)
Tests negative rejection

Compare fact-single questions with summary, reasoning, and unanswerable questions by answer composition and retrieval difficulty. section 14.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines direct-answer behavior, contrasts query types, and specifies grounded generation but stops short of answer inference.
Examples
Uses supply-voltage questions, extracted factual statements, and labeled context-query-reference triples.
What it assumes
Assumes a corpus and an LLM client, while teaching factual-statement extraction and question inversion from zero.
What you can do after
Build grounded fact-single evaluation triples and analyze their scores separately from harder query types.
How it connects
Connects fact-single questions to summary questions, reasoning questions, unanswerable questions, hybrid search, and retrieval evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.