BM25

As explained in The Art of AI Product Development

BM25 is a lexical ranking function that scores documents using term frequency, document frequency, and document-length normalization.

BM25 is a lexical ranking function for prioritizing documents with exact query terms. It scores term frequency, document frequency, and document-length normalization, complementing semantic search when meaning alone is too broad. section 7.2.4

Exact terms versus related meaning

A search for “augmented workflow” may retrieve broadly related documents through semantic search, while BM25 prioritizes documents containing that exact product phrase. The same hybrid approach helps retrieve API endpoints and distinguish “business” class from documents about doing business. section 7.2.4

Figure 7.8 Combining semantic and lexical search can increase overall retrieval accuracy.

Shows how combining semantic and lexical search can improve overall retrieval accuracy. section 7.2.4

How The Art of AI Product Development covers it

How deep it goes

Defines scoring through term frequency, document frequency, and length normalization without deriving the ranking algorithm.

Examples

Uses product taxonomy phrases, API endpoints, and the ambiguous term “business” to demonstrate exact-match needs.

What it assumes

Assumes semantic search is available and positions lexical ranking as its precision-oriented complement.

What you can do after

Apply BM25 through hybrid retrieval to prioritize exact terms while retaining semantically related context.

How it connects

Connects BM25 with lexical search, semantic search, embeddings, and retrieval accuracy.

liveBookmanning.com

Read The Art of AI Product Development on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Elasticsearch in Action, Second EditionOkapi BM25 is a probabilistic information-retrieval ranking algorithm that scores documents using term frequency, document frequency, and document-length normalization.
• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsA lexical information-retrieval ranking algorithm that scores documents using term frequency, inverse document frequency, and document length.
• 

  liveBookmanning.com
  Building Reliable AI SystemsBM25 is a keyword-based ranking algorithm that scores documents according to query-term frequency while accounting for document length and term saturation.
• 

  liveBookmanning.com
  Generative AI in ActionBM25 is a probabilistic ranking function that estimates document relevance to a query using term frequency, inverse document frequency, document length, and term-frequency saturation.

Compare how 5 books explain BM25 →

Related concepts

• lexical searchkind of · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
• hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
• similarity searchcontrasts with · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
• term frequency-inverse document frequencycontrasts with · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
• Elasticsearchused by · Elasticsearch is a distributed search and analytics engine that indexes data for searching, retrieval, and analysis.
• relevanceproduces · Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.

This explainer was generated by Manning's knowledge graph from the text of The Art of AI Product Development. The section links point to the parts of the book it draws on.