Natural Questions (NQ) is a benchmark built from real Google search queries, pairing naturally occurring questions with answers located inside full Wikipedia articles. It tests whether a system can answer from authentic information needs rather than supplied context. section 4.5 section 5.3
NQ exposes several open-domain question-answering requirements:
- Search broadly: systems may need to locate answers across 21 million Wikipedia passages. section 4.5
- Ground answers: dense passage retrieval finds relevant evidence, while generative systems can synthesize beyond verbatim spans. section 4.5
- Evaluate retrieval separately from answering: Recall@k measures whether evidence appears in the candidate set, while MRR rewards high ranking. section 3.2.1
- Test multi-document reasoning: performance can improve as systems process more retrieved passages, unlike extractive readers that may stagnate or decline. section 5.3
Questions represented in NQ
Examples include “What is the ozone layer and how does it affect Earth?”, “Who plays Voldemort in the Harry Potter movies?”, and “When did the Soviet Union collapse?” A system must search Wikipedia and generate or extract the precise answer from the retrieved evidence. section 4.5
