liveBook · Manning

Exact-match accuracy

As explained in Retrieval Augmented Generation, The Foundational Ideas

Exact Match is a question-answering metric that counts an answer as correct only when it exactly matches the reference answer after normalization.

Exact-match accuracy counts a prediction as correct only when its normalized answer matches the reference answer exactly. It is therefore an all-or-nothing measure: one differing answer string receives no credit. section 4.0 section 4.5.1

A strict answer check

For “When did humans first land on the moon?”, producing “1969” passes. A different answer fails, even when it reflects partial knowledge. In the Pacific Northwest example, “Seattle” is counted correct after retrieval combines population and descriptive evidence. section 4.5.1

Use exact-match accuracy to read several distinct evaluation signals:

  • Compare canonical-answer performance rather than partial similarity; F1 supplies partial credit for close or paraphrased answers. section 5.3
  • Interpret gains as more frequent production of the expected answer string, not merely vaguely improved wording. section 5.3
  • Use it to compare retrieval and generation systems on knowledge-intensive questions, including systems that synthesize multiple passages. section 4.0 section 4.5.1

The metric is demanding but operationally clear. A system can retrieve relevant information yet fail exact match if it cannot synthesize the canonical answer; conversely, a correct paraphrase may receive no credit. section 4.5.1 section 5.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines strict normalized matching and contrasts it with F1, but does not detail the normalization procedure.
Examples
Uses the answers “1969” and “Seattle,” plus TriviaQA and Natural Questions benchmark comparisons.
What it assumes
Assumes question-answering evaluation and reference answers while defining the distinction between exact match and F1.
What you can do after
Evaluate question-answering systems by checking canonical-answer accuracy and distinguishing exact gains from partial-overlap gains.
How it connects
Connects retrieval-augmented generation, dense retrieval, multi-document processing, extractive models, and F1.

Also explained in

Compare how 8 books explain exact-match accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.