liveBook · Manning

BM25

As explained in Elasticsearch in Action, Second Edition

Okapi BM25 is a probabilistic information-retrieval ranking algorithm that scores documents using term frequency, document frequency, and document-length normalization.

BM25 is Elasticsearch’s default similarity algorithm for full-text search. It assigns each matching document a positive floating-point _score, then returns higher-scoring documents first. The score reflects how well a document matches the query, helping a search for a book such as “Java” rank likely matches ahead of weaker ones. section 2.3.4 section 3.4.1

BM25 improves on TF-IDF in two practical ways. It accounts for term frequency, so a query term appearing in a document can raise relevance, but uses nonlinear saturation so repeated terms do not gain unlimited credit. It also normalizes for document length, reducing the advantage that longer documents might otherwise receive simply because they contain more words. section 3.4.2

Why short titles can rank higher

For a title search for “Java,” Effective Java can receive a higher score than Head First Java even though both match: the query word occupies a shorter field. A title with several occurrences, such as “Mastering Java: Learning Core Java and Enterprise Java With Examples,” may gain relevance from frequency, while BM25 prevents repetition alone from dominating the ranking. section 3.4.1 section 3.4.2

Table 3.5 Elasticsearch’s similarity algorithms (view table figure)

Similarity algorithm

Type

Description

Okapi BM25 (default)

BM25

An enhanced TF/IDF algorithm that considers field length in addition to term and document frequencies

Divergence from Randomness (DFR)

DFR

Uses the DFR framework developed by its authors, Amati and Rijsbergen, which aims to improve search relevance by measuring the divergence between the actual term and an expected random distribution. Terms that occur more often in relevant documents than in a random distribution are assigned higher weights when ranking search results.

Divergence from Independence (DFI)

DFI

A specific model of the DFR family that measures the divergence of the actual term frequency distribution from an independent distribution. DFI aims to assign higher scores to documents by comparing the observed term frequencies with those expected in a random, uncorrelated term frequency.

LM Dirichlet

LMDirichlet

Calculates the relevance of documents based on the probability of generating the query terms from the document’s language model

LM Jelinek-Mercer

LMJelinek-Mercer

Provides improved search result relevance compared to models that do not account for data sparsity

Manua

 

Creates a manual script

Boolean similarity

boolean

Does not consider ranking factors unless the query criteria are satisfied

Compare BM25 with Elasticsearch’s other built-in similarity algorithms when the default ranking behavior does not fit a field’s search requirements. section 3.4.2

Similarity is applied per field through mapping APIs. Elasticsearch can use alternatives to BM25 or custom similarity settings when needed; BM25 exposes k1 for nonlinear term-frequency saturation and b for document-length normalization, with defaults of 1.2 and 0.75. section 3.4.2

NOTE: A _score is relative relevance within a result set: the highest score is normally reported as max_score and belongs to the first result. Treat BM25 as a ranking mechanism for matching documents, then evaluate whether its ordering fits the search experience. section 3.4.1

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Explains term frequency, inverse document frequency, length normalization, saturation, parameters, and per-field selection, while deferring custom algorithms.
Examples
Uses Java book-title searches, contrasting Effective Java, Head First Java, and a title containing Java three times.
What it assumes
Introduces BM25 through Elasticsearch full-text queries and plain-language term and document frequency comparisons.
What you can do after
Operate/configure it by selecting similarity per field and recognizing how frequency and field length affect ranking.
How it connects
Connects BM25 with TF-IDF, relevance scores, field length, mappings, and Elasticsearch similarity algorithms.

Also explained in

Compare how 5 books explain BM25 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.