liveBook · Manning

Term frequency-inverse document frequency

As explained in LLM Evaluation and Alignment, The Foundational Ideas

TF-IDF is a lexical retrieval weighting scheme that combines a term’s frequency in a document with its inverse document frequency across a corpus.

TF-IDF is a sparse retrieval weighting scheme. It scores documents by combining how often query terms occur in each document with how rare those terms are across the corpus. The result is fast, interpretable keyword retrieval that emphasizes terminology without requiring semantic embeddings. section 6.2.1

Use TF-IDF retrieval through a simple indexing-and-ranking workflow:

  • Process and index the corpus into a searchable representation. section 6.2.1
  • Compare query terms with indexed documents, weighting frequent matches and corpus-specific terms more heavily. section 6.2.1
  • Rank documents and inspect precision@k, recall@k, or AP/MAP to determine whether relevant context reaches the retrieval cutoff. section 6.5 section 6.2.1

Domain-specific terminology

In a medical knowledge base, a document containing “myocardial infarction” can rank highly for a query using the same rare term. A query using “heart attack” may miss documents that use “MI” or “myocardial infarction,” because TF-IDF handles paraphrases poorly. section 6.2.1

Table 6.2 Retrieval approaches and their failure modes
Retrieval method
Strength
Failure modes
Useful diagnostic signals
Grep / ripgrep
Exact, deterministic matching
Lexical variation; exact matches that are not task relevant
Recall across term variations; precision of exact matches
Sparse / TF-IDF / BM25
Fast, interpretable, and strong on   terminology
Paraphrases and synonyms
precision@k, recall@k, AP/MAP
Dense / vector search
Semantic similarity, handles paraphrases
Precise identifiers, exact terms
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
Hybrid retrieval
Combines sparse and dense signals
Fusion weights, duplicate results, conflicting rankings
precision@k, recall@k, MAP, nDCG

Compare sparse TF-IDF retrieval with exact matching, dense vector search, and hybrid retrieval through their strengths, failure modes, and diagnostic metrics. section 6.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains term weighting, indexing, strengths, metrics, and paraphrase failure, while deferring formula-level implementation details.
Examples
Uses medical terminology such as “myocardial infarction,” “MI,” and “heart attack” to demonstrate matching and paraphrase gaps.
What it assumes
Requires a processed and indexed document corpus plus query terms suitable for lexical matching.
What you can do after
Evaluate sparse retrieval with precision@k, recall@k, and AP/MAP, then diagnose recall gaps caused by vocabulary mismatch.
How it connects
Connects TF-IDF to sparse retrieval, BM25, dense vector search, hybrid retrieval, precision, recall, and rank-aware metrics.

Also explained in

Compare how 12 books explain term frequency-inverse document frequency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.