Lexical search, also called sparse retrieval, ranks documents using statistical weights on query terms rather than requiring exact character matches. TF-IDF and BM25 combine term frequency with inverse document frequency; BM25 adds saturation and document-length normalization. section 6.2.1
The corpus must be processed and indexed before searching. Sparse retrieval is fast and interpretable, and needs neither embedding models nor GPUs at query time. It works well when domain terminology matters, but handles paraphrases poorly. section 6.2.1
Medical terminology
A query using “myocardial infarction” can rank documents containing that rare term highly. A query for “heart attack” may miss documents that use “MI” or “myocardial infarction.” section 6.2.1
Table 6.2 Retrieval approaches and their failure modes
|
Retrieval method
|
Strength
|
Failure modes
|
Useful diagnostic signals
|
|
Grep / ripgrep
|
Exact, deterministic matching
|
Lexical variation; exact matches that are not task relevant
|
Recall across term variations; precision of exact matches
|
|
Sparse / TF-IDF / BM25
|
Fast, interpretable, and strong on
terminology
|
Paraphrases and synonyms
|
precision@k, recall@k, AP/MAP
|
|
Dense / vector search
|
Semantic similarity, handles paraphrases
|
Precise identifiers, exact terms
|
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
|
|
Hybrid retrieval
|
Combines sparse and dense signals
|
Fusion weights, duplicate results, conflicting rankings
|
precision@k, recall@k, MAP, nDCG
|
Compare sparse retrieval with exact matching, dense search, and hybrid retrieval, including their failure modes and diagnostic metrics. section 6.2.1













