liveBook · Manning

Exact-match accuracy

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An exact-match heuristic detects possible data overlap by checking whether identical token or text sequences occur in two data sources.

Exact-match accuracy checks whether identical token, character, n-gram, or text spans occur in two data sources. It provides a deterministic match/no-match signal, but an exact match may be irrelevant and a non-match may still be semantically related. section 6.2.1 section 2.3.4

A practical overlap check typically proceeds as follows:

  • Select the units to compare, such as hashed n-grams, substrings, or longer spans. section 2.3.4
  • Index or hash the reference corpus, then search evaluation examples for identical units. section 2.3.4
  • Inspect matches in context rather than treating the count as proof of contamination; constrained formats and canonical phrasing can produce naturally high overlap. section 2.3.4

Benchmark leakage

A GPT-3 evaluation protocol created cleaner benchmark subsets by removing examples with 13-gram overlap against pretraining data. The same approach can flag duplicate or templated examples inside curated datasets. section 2.3.4

Table 6.2 Retrieval approaches and their failure modes
Retrieval method
Strength
Failure modes
Useful diagnostic signals
Grep / ripgrep
Exact, deterministic matching
Lexical variation; exact matches that are not task relevant
Recall across term variations; precision of exact matches
Sparse / TF-IDF / BM25
Fast, interpretable, and strong on   terminology
Paraphrases and synonyms
precision@k, recall@k, AP/MAP
Dense / vector search
Semantic similarity, handles paraphrases
Precise identifiers, exact terms
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
Hybrid retrieval
Combines sparse and dense signals
Fusion weights, duplicate results, conflicting rankings
precision@k, recall@k, MAP, nDCG

Compare exact matching with sparse, dense, and hybrid retrieval to separate lexical overlap from semantic retrieval and identify likely failure modes. section 6.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines binary matching and describes n-gram, substring, span, hashing, and indexing methods, but stops short of semantic similarity.
Examples
Uses grep-style searches, 13-gram benchmark filtering, duplicate auditing, and verbatim reference-answer overlap.
What it assumes
Assumes familiarity with datasets and text sequences, while defining contamination, overlap units, and interpretation caveats.
What you can do after
Diagnose lexical retrieval gaps and investigate possible benchmark contamination using exact overlap checks.
How it connects
Connects exact matching with data contamination, n-gram overlap, lexical overlap, sparse retrieval, and semantic retrieval.

Also explained in

Compare how 8 books explain exact-match accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.