TF-IDF is a sparse retrieval weighting scheme. It scores documents by combining how often query terms occur in each document with how rare those terms are across the corpus. The result is fast, interpretable keyword retrieval that emphasizes terminology without requiring semantic embeddings. section 6.2.1
Use TF-IDF retrieval through a simple indexing-and-ranking workflow:
- Process and index the corpus into a searchable representation. section 6.2.1
- Compare query terms with indexed documents, weighting frequent matches and corpus-specific terms more heavily. section 6.2.1
- Rank documents and inspect precision@k, recall@k, or AP/MAP to determine whether relevant context reaches the retrieval cutoff. section 6.5 section 6.2.1
Domain-specific terminology
In a medical knowledge base, a document containing “myocardial infarction” can rank highly for a query using the same rare term. A query using “heart attack” may miss documents that use “MI” or “myocardial infarction,” because TF-IDF handles paraphrases poorly. section 6.2.1
Table 6.2 Retrieval approaches and their failure modes
|
Retrieval method
|
Strength
|
Failure modes
|
Useful diagnostic signals
|
|
Grep / ripgrep
|
Exact, deterministic matching
|
Lexical variation; exact matches that are not task relevant
|
Recall across term variations; precision of exact matches
|
|
Sparse / TF-IDF / BM25
|
Fast, interpretable, and strong on
terminology
|
Paraphrases and synonyms
|
precision@k, recall@k, AP/MAP
|
|
Dense / vector search
|
Semantic similarity, handles paraphrases
|
Precise identifiers, exact terms
|
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
|
|
Hybrid retrieval
|
Combines sparse and dense signals
|
Fusion weights, duplicate results, conflicting rankings
|
precision@k, recall@k, MAP, nDCG
|
Compare sparse TF-IDF retrieval with exact matching, dense vector search, and hybrid retrieval through their strengths, failure modes, and diagnostic metrics. section 6.2.1











