A reasoning question requires an answer inferred from context rather than copied from a stated fact. It may involve interpreting trends, comparing policies, or combining evidence across passages. section 14.4
Use reasoning questions to test the hardest retrieval and generation behaviors:
- Ask for a conclusion that is not stated verbatim, such as whether consumption increased over five years. section 14.4
- Separate them from fact_single questions, which retrieve one directly stated fact, and summary questions, which combine several contextual details. section 14.4
- Include them in balanced evaluation sets because fact-heavy benchmarks can make retrieval systems appear strong while missing common enterprise queries. section 14.4 section 14.7
- Tune retrieval against the question mix: the effective hybrid-search text weight can differ substantially between fact_single, summary, and reasoning queries. section 14.4
Compliance-assistant evaluation
A loan-advisor assistant may retrieve current and superseded policy documents for a question about reapplication eligibility. Better retrieval can expose contradictory clauses, after which generation may blend conditions from separate sections. Per-claim faithfulness checks reveal this failure even when context relevance improves. section 14.6
Table 14.3 Question-context taxonomy for RAG evaluation datasets
|
Class
|
Description
|
Example query
|
Retrieval difficulty
|
|
fact_single
|
Answer is one unit of information, directly stated in
context
|
"What supply voltage should I use?"
|
Easiest: direct lexical/semantic match
|
|
summary
|
Answer requires multiple pieces from context
|
"Summarize the key findings."
|
Medium: must retrieve the broad context
|
|
reasoning
|
Answer must be inferred, not directly stated
|
"Has consumption increased over 5 years?"
|
Hardest: context doesn't contain the answer verbatim
|
|
unanswerable
|
Answer cannot be derived from the context
|
"Is a tomato a fruit or a vegetable?" (asked of
a coffee machine patent)
|
Tests negative rejection
|
Compare reasoning with fact_single, summary, and unanswerable questions by answer derivability and retrieval difficulty. section 14.4
Listing 14.4 Balanced synthetic evaluation dataset generation.
def generate_eval_triples(
llm_client, passages: list[str], n_per_type: int = 5
) -> list[dict]:
"""Generate balanced Q&A pairs from corpus passages."""
triples = [] #A
for passage in passages:
facts = llm_client.generate(
f"Extract 5 factual statements from this "
f"text, one per line:\n{passage}"
).strip().split("\n")
for fact in facts[:n_per_type]: #B
question = llm_client.generate(
f"Write a specific question that this "
f"statement answers:\n{fact}"
)
triples.append({
"context": passage,
"query": question.strip(),
"reference_answer": fact.strip(),
"label": "fact_single",
})
summary_q = llm_client.generate( #C
f"Write a question requiring a summary of "
f"the main points in this text:\n{passage}"
)
triples.append({
"context": passage,
"query": summary_q.strip(),
"reference_answer": "Requires multi-fact summary",
"label": "summary",
})
reasoning_q = llm_client.generate( #D
f"Write a question that requires inference "
f"or reasoning about this text (the answer "
f"should not be directly stated):\n{passage}"
)
triples.append({
"context": passage,
"query": reasoning_q.strip(),
"reference_answer": "Requires reasoning",
"label": "reasoning",
})
return triples #EGenerate grounded reasoning questions by extracting information from corpus passages first, then prompting for targeted question types and labels. section 14.4.1
