liveBook · Manning

Lexical search

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Sparse retrieval ranks documents using statistical weights for query terms, typically with sparse keyword representations.

Lexical search, also called sparse retrieval, ranks documents using statistical weights on query terms rather than requiring exact character matches. TF-IDF and BM25 combine term frequency with inverse document frequency; BM25 adds saturation and document-length normalization. section 6.2.1

The corpus must be processed and indexed before searching. Sparse retrieval is fast and interpretable, and needs neither embedding models nor GPUs at query time. It works well when domain terminology matters, but handles paraphrases poorly. section 6.2.1

Medical terminology

A query using “myocardial infarction” can rank documents containing that rare term highly. A query for “heart attack” may miss documents that use “MI” or “myocardial infarction.” section 6.2.1

Table 6.2 Retrieval approaches and their failure modes
Retrieval method
Strength
Failure modes
Useful diagnostic signals
Grep / ripgrep
Exact, deterministic matching
Lexical variation; exact matches that are not task relevant
Recall across term variations; precision of exact matches
Sparse / TF-IDF / BM25
Fast, interpretable, and strong on   terminology
Paraphrases and synonyms
precision@k, recall@k, AP/MAP
Dense / vector search
Semantic similarity, handles paraphrases
Precise identifiers, exact terms
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
Hybrid retrieval
Combines sparse and dense signals
Fusion weights, duplicate results, conflicting rankings
precision@k, recall@k, MAP, nDCG

Compare sparse retrieval with exact matching, dense search, and hybrid retrieval, including their failure modes and diagnostic metrics. section 6.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains term-frequency and inverse-document-frequency scoring, BM25 saturation, document-length normalization, and corpus indexing.
Examples
Uses medical queries involving “myocardial infarction,” “heart attack,” and “MI” to illustrate terminology mismatch.
What you can do after
Evaluate retrieval with precision@k, recall@k, and AP/MAP while checking for paraphrase failures.
How it connects
Connects sparse retrieval to TF-IDF, BM25, pattern matching, dense retrieval, and hybrid retrieval.

Also explained in

Compare how 14 books explain lexical search →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.