liveBook · Manning

BM25

As explained in LLM Evaluation and Alignment, The Foundational Ideas

BM25 is a probabilistic ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.

BM25 (Best Match 25) is a probabilistic ranking function that scores documents using term frequency, inverse document frequency, and document-length normalization. It is widely used in production search systems and RAG pipelines. section 6.2.1

Use BM25 as a sparse retrieval method when terminology matters:

  • Prefer it for fast, interpretable retrieval with strong performance on precise terminology. section 6.2.1
  • Expect failures when relevant documents use paraphrases or synonyms rather than the query’s terms. section 6.2.1 section 6.5
  • Evaluate retrieval with precision@k, recall@k, and AP or MAP, especially when ranking affects the context-window cutoff. section 6.2.1 section 6.5

Choosing retrieval by query type

An agent can select a BM25 index for a query involving precise terminology, while using dense search for conceptual questions. Comparing the resulting precision and recall helps expose each method’s failure mode. section 6.2.1 section 6.5

Table 6.2 Retrieval approaches and their failure modes
Retrieval method
Strength
Failure modes
Useful diagnostic signals
Grep / ripgrep
Exact, deterministic matching
Lexical variation; exact matches that are not task relevant
Recall across term variations; precision of exact matches
Sparse / TF-IDF / BM25
Fast, interpretable, and strong on   terminology
Paraphrases and synonyms
precision@k, recall@k, AP/MAP
Dense / vector search
Semantic similarity, handles paraphrases
Precise identifiers, exact terms
recall@k, nDCG with graded relevance, hybrid-vs-dense comparisons
Hybrid retrieval
Combines sparse and dense signals
Fusion weights, duplicate results, conflicting rankings
precision@k, recall@k, MAP, nDCG

Compare BM25 with exact matching, dense vector search, and hybrid retrieval through their strengths, failure modes, and diagnostic metrics. section 6.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines BM25 through its scoring factors and identifies its strengths, failure modes, and evaluation metrics without deriving the formula.
Examples
Contrasts precise-terminology queries with conceptual or paraphrased questions to show when BM25 or dense search is preferable.
What you can do after
Diagnose retrieval failures, choose BM25 for terminology-heavy queries, and evaluate rankings with precision, recall, and MAP.
How it connects
Connects BM25 to TF-IDF, sparse retrieval, dense vector search, hybrid retrieval, and rank-aware retrieval metrics.

Also explained in

Compare how 7 books explain BM25 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.