An unanswerable question asks for information that the supplied context or knowledge base does not support. It tests whether a system recognizes missing information instead of fabricating an answer. section 14.4.1
Use unanswerable questions to strengthen evaluation coverage:
- Generate questions from passages in unrelated corpus sections, then pair each question with the original passage as context. section 14.4.1
- Label these cases separately so evaluation can reveal failures involving missing content rather than hiding them in aggregate scores. section 14.4.1
- Include synthetic cases alongside captured production interactions and expert-authored edge cases. section 14.4.2
Testing knowledge boundaries
Take a passage from one corpus section, generate a question from unrelated content, and retain the first passage as context. A reliable system should recognize that the context lacks sufficient information rather than inventing an answer. section 14.4.1
Listing 14.4 Balanced synthetic evaluation dataset generation.
def generate_eval_triples(
llm_client, passages: list[str], n_per_type: int = 5
) -> list[dict]:
"""Generate balanced Q&A pairs from corpus passages."""
triples = [] #A
for passage in passages:
facts = llm_client.generate(
f"Extract 5 factual statements from this "
f"text, one per line:\n{passage}"
).strip().split("\n")
for fact in facts[:n_per_type]: #B
question = llm_client.generate(
f"Write a specific question that this "
f"statement answers:\n{fact}"
)
triples.append({
"context": passage,
"query": question.strip(),
"reference_answer": fact.strip(),
"label": "fact_single",
})
summary_q = llm_client.generate( #C
f"Write a question requiring a summary of "
f"the main points in this text:\n{passage}"
)
triples.append({
"context": passage,
"query": summary_q.strip(),
"reference_answer": "Requires multi-fact summary",
"label": "summary",
})
reasoning_q = llm_client.generate( #D
f"Write a question that requires inference "
f"or reasoning about this text (the answer "
f"should not be directly stated):\n{passage}"
)
triples.append({
"context": passage,
"query": reasoning_q.strip(),
"reference_answer": "Requires reasoning",
"label": "reasoning",
})
return triples #EListing 14.4 provides a corpus-driven pipeline for generating labeled evaluation triples, which can be extended with unanswerable cases. section 14.4.1
