liveBook · Manning

Natural Questions

As explained in Retrieval Augmented Generation, The Foundational Ideas

Natural Questions is a dataset of naturally occurring questions paired with answers for training and evaluating question-answering systems.

Natural Questions (NQ) is a benchmark built from real Google search queries, pairing naturally occurring questions with answers located inside full Wikipedia articles. It tests whether a system can answer from authentic information needs rather than supplied context. section 4.5 section 5.3

NQ exposes several open-domain question-answering requirements:

  • Search broadly: systems may need to locate answers across 21 million Wikipedia passages. section 4.5
  • Ground answers: dense passage retrieval finds relevant evidence, while generative systems can synthesize beyond verbatim spans. section 4.5
  • Evaluate retrieval separately from answering: Recall@k measures whether evidence appears in the candidate set, while MRR rewards high ranking. section 3.2.1
  • Test multi-document reasoning: performance can improve as systems process more retrieved passages, unlike extractive readers that may stagnate or decline. section 5.3

Questions represented in NQ

Examples include “What is the ozone layer and how does it affect Earth?”, “Who plays Voldemort in the Harry Potter movies?”, and “When did the Soviet Union collapse?” A system must search Wikipedia and generate or extract the precise answer from the retrieved evidence. section 4.5

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines the dataset and evaluation setting, then compares retrieval, generation, and multi-document reasoning without detailing dataset construction procedures.
Examples
Uses questions about the ozone layer, Voldemort, and the Soviet Union alongside Wikipedia retrieval and multi-passage reasoning examples.
What you can do after
Evaluate retrieval and answering systems using evidence coverage, ranking quality, exactness, and performance as retrieved passages increase.
How it connects
Connects to open-domain question answering, retrieval-augmented generation, dense passage retrieval, Fusion-in-Decoder, Recall@k, MRR, Exact Match, and F1.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.