BM25
BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
Elasticsearch in Action, Second Edition
Okapi BM25 is a probabilistic information-retrieval ranking algorithm that scores documents using term frequency, document frequency, and document-length normalization.
- How it is used
- Ranks full-text Elasticsearch results by a relevance score so stronger document matches appear first.
- How deep it goes
- Explains term frequency, inverse document frequency, length normalization, saturation, parameters, and per-field selection, while deferring custom algorithms.
- Examples
- Uses Java book-title searches, contrasting Effective Java, Head First Java, and a title containing Java three times.
- What you can do after
- Operate/configure it by selecting similarity per field and recognizing how frequency and field length affect ranking.
Read how Elasticsearch in Action, Second Edition explains BM25 →
Domain-Specific Small Language Models
A lexical information-retrieval ranking algorithm that scores documents using term frequency, inverse document frequency, and document length.
- How it is used
- Serves as the keyword-retrieval baseline inside agentic RAG and hybrid search workflows.
- How deep it goes
- Defines BM25 at a ranking-formula level and stops short of derivation or implementation details.
- Examples
- Uses local document search, a Smart Home technologies query, and a hybrid local LanceDB search example.
- What you can do after
- Apply it in retrieval tools to rank local documents by keyword relevance.
Read how Domain-Specific Small Language Models explains BM25 →
The Art of AI Product Development
BM25 is a lexical ranking function that scores documents using term frequency, document frequency, and document-length normalization.
- How it is used
- Deploys BM25 alongside semantic search to improve precision for technical, product-specific, and ambiguous queries.
- How deep it goes
- Defines scoring through term frequency, document frequency, and length normalization without deriving the ranking algorithm.
- Examples
- Uses product taxonomy phrases, API endpoints, and the ambiguous term “business” to demonstrate exact-match needs.
- What you can do after
- Apply BM25 through hybrid retrieval to prioritize exact terms while retaining semantically related context.
Building Reliable AI Systems
BM25 is a keyword-based ranking algorithm that scores documents according to query-term frequency while accounting for document length and term saturation.
- How it is used
- Ranks documents by query-term frequency as a fast, reproducible keyword-retrieval baseline.
- How deep it goes
- Explains term-frequency ranking and common-word penalties qualitatively, without deriving a scoring formula.
- Examples
- Uses an iPhone troubleshooting search and a telecom roaming query with device and plan details.
- What you can do after
- Apply BM25 retrieval with Haystack's InMemoryBM25Retriever and connect extracted metadata filters to the retriever.
Generative AI in Action
BM25 is a probabilistic ranking function that estimates document relevance to a query using term frequency, inverse document frequency, document length, and term-frequency saturation.
- How it is used
- Ranks documents for retrieval-augmented generation and search queries where lexical matching and efficiency are important.
- How deep it goes
- Defines ranking inputs and contrasts alternatives but stops short of mathematical derivation or implementation details.
- Examples
- Uses query-term frequency, inverse document frequency, document length, and term-frequency saturation as ranking signals.
- What you can do after
- Evaluate/understand retriever choices using lexical matching, semantic understanding, computational efficiency, and resource availability.
Related concepts
- lexical searchkind of · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
- hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
- similarity searchcontrasts with · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
- term frequency-inverse document frequencycontrasts with · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
- Elasticsearchused by · Elasticsearch is a distributed search and analytics engine that indexes data for searching, retrieval, and analysis.
- relevanceproduces · Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.