liveBook · Manning

Reasoning question

As explained in Retrieval Augmented Generation, The Foundational Ideas

A reasoning question or query requires interpretation, comparison, inference, or logical combination beyond direct extraction of a stated fact.

A reasoning question requires an answer inferred from context rather than copied from a stated fact. It may involve interpreting trends, comparing policies, or combining evidence across passages. section 14.4

Use reasoning questions to test the hardest retrieval and generation behaviors:

  • Ask for a conclusion that is not stated verbatim, such as whether consumption increased over five years. section 14.4
  • Separate them from fact_single questions, which retrieve one directly stated fact, and summary questions, which combine several contextual details. section 14.4
  • Include them in balanced evaluation sets because fact-heavy benchmarks can make retrieval systems appear strong while missing common enterprise queries. section 14.4 section 14.7
  • Tune retrieval against the question mix: the effective hybrid-search text weight can differ substantially between fact_single, summary, and reasoning queries. section 14.4

Compliance-assistant evaluation

A loan-advisor assistant may retrieve current and superseded policy documents for a question about reapplication eligibility. Better retrieval can expose contradictory clauses, after which generation may blend conditions from separate sections. Per-claim faithfulness checks reveal this failure even when context relevance improves. section 14.6

Table 14.3 Question-context taxonomy for RAG evaluation datasets
Class
Description
Example query
Retrieval difficulty
fact_single
Answer is one unit of information, directly stated in context
"What supply voltage should I use?"
Easiest: direct lexical/semantic match
summary
Answer requires multiple pieces from context
"Summarize the key findings."
Medium: must retrieve the broad context
reasoning
Answer must be inferred, not directly stated
"Has consumption increased over 5 years?"
Hardest: context doesn't contain the answer verbatim
unanswerable
Answer cannot be derived from the context
"Is a tomato a fruit or a vegetable?" (asked of a coffee machine patent)
Tests negative rejection

Compare reasoning with fact_single, summary, and unanswerable questions by answer derivability and retrieval difficulty. section 14.4

Listing 14.4 Balanced synthetic evaluation dataset generation.
def generate_eval_triples(
    llm_client, passages: list[str], n_per_type: int = 5
) -> list[dict]:
    """Generate balanced Q&A pairs from corpus passages."""
    triples = []  #A
    for passage in passages:
        facts = llm_client.generate(
            f"Extract 5 factual statements from this "
            f"text, one per line:\n{passage}"
        ).strip().split("\n")
        for fact in facts[:n_per_type]:  #B
            question = llm_client.generate(
                f"Write a specific question that this "
                f"statement answers:\n{fact}"
            )
            triples.append({
                "context": passage,
                "query": question.strip(),
                "reference_answer": fact.strip(),
                "label": "fact_single",
            })
        summary_q = llm_client.generate(  #C
            f"Write a question requiring a summary of "
            f"the main points in this text:\n{passage}"
        )
        triples.append({
            "context": passage,
            "query": summary_q.strip(),
            "reference_answer": "Requires multi-fact summary",
            "label": "summary",
        })
        reasoning_q = llm_client.generate(  #D
            f"Write a question that requires inference "
            f"or reasoning about this text (the answer "
            f"should not be directly stated):\n{passage}"
        )
        triples.append({
            "context": passage,
            "query": reasoning_q.strip(),
            "reference_answer": "Requires reasoning",
            "label": "reasoning",
        })
    return triples  #E

Generate grounded reasoning questions by extracting information from corpus passages first, then prompting for targeted question types and labels. section 14.4.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines reasoning questions, contrasts their retrieval difficulty, and shows dataset-generation and production-diagnosis procedures.
Examples
Uses trend interpretation, policy comparison, and compliance-eligibility queries as concrete reasoning-question examples.
What it assumes
Assumes familiarity with passages, retrieval, RAG evaluation sets, and direct fact questions.
What you can do after
Evaluate RAG systems with balanced reasoning coverage and diagnose retrieval or faithfulness failures exposed by inferred-answer queries.
How it connects
Connects reasoning questions to RAG evaluation, hybrid search, retrieval, faithfulness, context relevance, and unanswerable questions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.