BM25

As explained in Domain-Specific Small Language Models

A lexical information-retrieval ranking algorithm that scores documents using term frequency, inverse document frequency, and document length.

BM25 ranks documents for keyword-based retrieval using term frequency, inverse document frequency, and document length. It appears as a local retrieval option inside agentic RAG workflows. section 14.2 section 14.4

Used here alongside adjacent retrieval and agent patterns:

• Pairs with vector search in hybrid retrieval. section 14.4
• Feeds CodeAgent tool choice in agentic RAG. section 14.2
• Contrasts with semantic search over embeddings. section 14.2

Search tool over local documents

A BM25 search tool wraps a local document list, invokes BM25Retriever.from_documents(docs, k=3), and returns the retrieved document text for a query. section 14.2

Hybrid local search

A hybrid tool combines BM25 keyword retrieval with LanceDB vector search, then lets CodeAgent choose the local search strategy for a query. section 14.2 section 14.4

Figure 14.3 The logical schema of a basic agentic RAG system

Shows the basic agentic RAG schema where retrieval happens before generation and can fall back to web search if local results are weak. section 14.2

How Domain-Specific Small Language Models covers it

How deep it goes

Defines BM25 at a ranking-formula level and stops short of derivation or implementation details.

Examples

Uses local document search, a Smart Home technologies query, and a hybrid local LanceDB search example.

What it assumes

Assumes retrieval concepts and document chunking, while introducing BM25 as a named keyword-ranking method.

What you can do after

Apply it in retrieval tools to rank local documents by keyword relevance.

How it connects

Links to agentic RAG, semantic search, hybrid search, CodeAgent, and EnsembleRetriever.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Elasticsearch in Action, Second EditionOkapi BM25 is a probabilistic information-retrieval ranking algorithm that scores documents using term frequency, document frequency, and document-length normalization.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentBM25 is a lexical ranking function that scores documents using term frequency, document frequency, and document-length normalization.
• 

  liveBookmanning.com
  Building Reliable AI SystemsBM25 is a keyword-based ranking algorithm that scores documents according to query-term frequency while accounting for document length and term saturation.
• 

  liveBookmanning.com
  Generative AI in ActionBM25 is a probabilistic ranking function that estimates document relevance to a query using term frequency, inverse document frequency, document length, and term-frequency saturation.

Compare how 5 books explain BM25 →

Related concepts

• lexical searchkind of · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
• hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
• similarity searchcontrasts with · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
• term frequency-inverse document frequencycontrasts with · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
• Elasticsearchused by · Elasticsearch is a distributed search and analytics engine that indexes data for searching, retrieval, and analysis.
• relevanceproduces · Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.