Exact-match accuracy checks whether identical token, character, n-gram, or text spans occur in two data sources. It provides a deterministic match/no-match signal, but an exact match may be irrelevant and a non-match may still be semantically related. section 6.2.1 section 2.3.4
A practical overlap check typically proceeds as follows:
- Select the units to compare, such as hashed n-grams, substrings, or longer spans. section 2.3.4
- Index or hash the reference corpus, then search evaluation examples for identical units. section 2.3.4
- Inspect matches in context rather than treating the count as proof of contamination; constrained formats and canonical phrasing can produce naturally high overlap. section 2.3.4
Benchmark leakage
A GPT-3 evaluation protocol created cleaner benchmark subsets by removing examples with 13-gram overlap against pretraining data. The same approach can flag duplicate or templated examples inside curated datasets. section 2.3.4
Table 6.2 Retrieval approaches and their failure modes
|
Retrieval method
|
Strength
|
Failure modes
|
Useful diagnostic signals
|
|
Grep / ripgrep
|
Exact, deterministic matching
|
Lexical variation; exact matches that are not task relevant
|
Recall across term variations; precision of exact matches
|
|
Sparse / TF-IDF / BM25
|
Fast, interpretable, and strong on
terminology
|
Paraphrases and synonyms
|
precision@k, recall@k, AP/MAP
|
|
Dense / vector search
|
Semantic similarity, handles paraphrases
|
Precise identifiers, exact terms
|
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
|
|
Hybrid retrieval
|
Combines sparse and dense signals
|
Fusion weights, duplicate results, conflicting rankings
|
precision@k, recall@k, MAP, nDCG
|
Compare exact matching with sparse, dense, and hybrid retrieval to separate lexical overlap from semantic retrieval and identify likely failure modes. section 6.2.1







