A fact-single question asks for one factual item stated directly in the supplied context. For example, “What supply voltage should I use?” can be answered by locating a single matching statement rather than combining evidence or inferring a conclusion. section 14.4
Fact-single questions are useful for testing direct retrieval, but they can make a RAG system appear stronger than it is. Public QA benchmarks commonly contain 60–80% of them, while enterprise users often ask summary and reasoning questions. The best hybrid-search text weight can also change by question type, so a fact-single-heavy test set may tune retrieval for the wrong workload. section 14.4 section 14.7
Use fact-single questions to:
- Check whether retrieval finds the passage containing an explicit answer. section 14.4
- Generate grounded evaluation pairs by extracting factual statements first, then reversing each statement into a question. section 14.4.1
- Label results separately from summary, reasoning, and unanswerable questions so evaluation scores reveal query-type weaknesses. section 14.4 section 14.4.1
Grounded generation pattern
Given a corpus passage, extract an individual statement such as a specification or requirement. Ask an LLM to write a specific question that the statement answers, and store the passage, question, statement as the reference answer, and the label “fact_single.” This keeps the answer grounded in actual corpus content and reduces hallucination in the evaluation data. section 14.4.1
Table 14.3 Question-context taxonomy for RAG evaluation datasets
|
Class
|
Description
|
Example query
|
Retrieval difficulty
|
|
fact_single
|
Answer is one unit of information, directly stated in
context
|
"What supply voltage should I use?"
|
Easiest: direct lexical/semantic match
|
|
summary
|
Answer requires multiple pieces from context
|
"Summarize the key findings."
|
Medium: must retrieve the broad context
|
|
reasoning
|
Answer must be inferred, not directly stated
|
"Has consumption increased over 5 years?"
|
Hardest: context doesn't contain the answer verbatim
|
|
unanswerable
|
Answer cannot be derived from the context
|
"Is a tomato a fruit or a vegetable?" (asked of
a coffee machine patent)
|
Tests negative rejection
|
Compare fact-single questions with summary, reasoning, and unanswerable questions by answer composition and retrieval difficulty. section 14.4
