liveBook · Manning

Unanswerable question

As explained in Retrieval Augmented Generation, The Foundational Ideas

An unanswerable question is an evaluation question whose answer is not supported by the system's knowledge base or supplied context.

An unanswerable question asks for information that the supplied context or knowledge base does not support. It tests whether a system recognizes missing information instead of fabricating an answer. section 14.4.1

Use unanswerable questions to strengthen evaluation coverage:

  • Generate questions from passages in unrelated corpus sections, then pair each question with the original passage as context. section 14.4.1
  • Label these cases separately so evaluation can reveal failures involving missing content rather than hiding them in aggregate scores. section 14.4.1
  • Include synthetic cases alongside captured production interactions and expert-authored edge cases. section 14.4.2

Testing knowledge boundaries

Take a passage from one corpus section, generate a question from unrelated content, and retain the first passage as context. A reliable system should recognize that the context lacks sufficient information rather than inventing an answer. section 14.4.1

Listing 14.4 Balanced synthetic evaluation dataset generation.
def generate_eval_triples(
    llm_client, passages: list[str], n_per_type: int = 5
) -> list[dict]:
    """Generate balanced Q&A pairs from corpus passages."""
    triples = []  #A
    for passage in passages:
        facts = llm_client.generate(
            f"Extract 5 factual statements from this "
            f"text, one per line:\n{passage}"
        ).strip().split("\n")
        for fact in facts[:n_per_type]:  #B
            question = llm_client.generate(
                f"Write a specific question that this "
                f"statement answers:\n{fact}"
            )
            triples.append({
                "context": passage,
                "query": question.strip(),
                "reference_answer": fact.strip(),
                "label": "fact_single",
            })
        summary_q = llm_client.generate(  #C
            f"Write a question requiring a summary of "
            f"the main points in this text:\n{passage}"
        )
        triples.append({
            "context": passage,
            "query": summary_q.strip(),
            "reference_answer": "Requires multi-fact summary",
            "label": "summary",
        })
        reasoning_q = llm_client.generate(  #D
            f"Write a question that requires inference "
            f"or reasoning about this text (the answer "
            f"should not be directly stated):\n{passage}"
        )
        triples.append({
            "context": passage,
            "query": reasoning_q.strip(),
            "reference_answer": "Requires reasoning",
            "label": "reasoning",
        })
    return triples  #E

Listing 14.4 provides a corpus-driven pipeline for generating labeled evaluation triples, which can be extended with unanswerable cases. section 14.4.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains corpus-based construction and diagnostic purpose, without detailing runtime refusal behavior.
Examples
Uses unrelated corpus passages, synthetic questions, captured production interactions, and expert-authored edge cases.
What it assumes
Assumes a corpus and evaluation pipeline, while deriving questions from corpus content rather than requiring manual annotation.
What you can do after
Evaluate system awareness of knowledge-base boundaries with balanced test data.
How it connects
Connects to retrieval-augmented generation, evaluation datasets, missing content, hallucination, and regression testing.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.