liveBook · Manning

BM25

As explained in Retrieval Augmented Generation, The Foundational Ideas

BM25 is a probabilistic term-weighting and document-ranking function based on term frequency, inverse document frequency, and document-length normalization.

BM25 is a probabilistic ranking function for keyword search. It scores documents using query-term frequency, term importance across the corpus, document length, and the hyperparameters k1 and b. section 2.1.1

BM25 is useful when exact lexical matching is the intended signal, but it has clear boundaries:

  • Weights rarer query terms more heavily and controls the effect of repeated terms through term-frequency saturation. section 2.1.1 section 2.1.3
  • Normalizes for document length, helping compare documents without treating long documents as automatically more relevant. section 2.1.1
  • Matches terms discretely: automobile, car, and vehicle remain unrelated unless preprocessing adds synonym expansion, lemmatization, or typo correction. section 2.1.1 section 2.1.3

Keyword mismatch in a toy corpus

For the query “automobile maintenance,” BM25 ranks “The automobile industry is booming” first with 1.17 and gives “I bought a new car today” a score of 0.00 because it shares no query terms. Documents containing maintenance or automobile receive nonzero scores, while semantic similarity is not inferred. section 2.1.3

s(q,d)=∑i=1nIDF(qi)f(qi,d)(k1+1)f(qi,d) + k1 (1-b+b|d|l),

Inspect the BM25 scoring equation, including inverse document frequency, term frequency, document length, average length, k1, and b. section 2.1.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the scoring variables, frequency saturation, and length normalization, but stops short of semantic matching.
Examples
Uses a five-document corpus comparing automobile, car, vehicle, maintenance, and unrelated computer terms.
What it assumes
Assumes familiarity with documents, queries, term frequency, and inverse document frequency while defining BM25 variables.
What you can do after
Apply BM25 via the rank_bm25 Python implementation and interpret exact-term ranking failures.
How it connects
Connects BM25 to TF-IDF, Word2Vec, semantic search, synonym expansion, lemmatization, and typo correction.

Also explained in

Compare how 7 books explain BM25 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.