Similarity scoring in Elasticsearch in Action, Second Edition

Q: How deeply does Elasticsearch in Action, Second Edition cover similarity scoring?

Explains BM25 and TF-IDF through term frequency, rarity, and length normalization; defers custom similarity algorithms.

A similarity algorithm computes how closely a document matches a query to support relevance scoring.

Similarity scoring ranks full-text search results by how closely each document matches a query. Elasticsearch attaches a positive floating-point _score to each hit and returns higher-scored documents first. For a search such as Java in book titles, a result with a stronger match appears above weaker matches. section 1.2.1 section 3.4.1

Elasticsearch uses BM25 by default. BM25 considers term frequency, inverse document frequency, and field length. A term that occurs often in one document can help its score, while a term common across the collection is less distinctive. Field-length normalization also prevents longer documents from receiving undue credit merely because they contain more words. section 3.4.1 section 3.4.2

The main scoring ideas are:

Term frequency (TF): repeated occurrences of a query term can increase relevance. section 3.4.2
Inverse document frequency (IDF): terms that are rarer across the document collection carry more relevance. section 3.4.2
Field length: the same term can be more meaningful in a short title than in a long synopsis. section 3.4.1 section 3.4.2
Term-frequency saturation: BM25 limits the benefit of excessive repetition. section 3.4.2

Rank a bookstore search across title and synopsis

A multi_match query searches both fields for Java. Elasticsearch calculates _score values with BM25 and sorts the returned books accordingly. A title such as Effective Java can rank above Head First Java when the matching word occurs in a shorter field, even though both titles contain Java. GET books/_search { "query": { "multi_match": { "query": "Java", "fields": ["title", "synopsis"] } } } section 2.3.4 section 3.4.1

Use scoring when result order should reflect relevance. Query-context clauses, including must and should, produce scores; filter-context clauses do not, avoiding scoring work when only matching is needed. A should clause can raise the score of documents that satisfy additional preferred criteria. section 11.3.6 section 11.3.6

BM25 is usually an effective default, but similarity can be selected per field through mapping APIs when fields have different search behavior. Elasticsearch also provides other built-in similarity algorithms. Custom algorithms are available but are an advanced capability. section 3.4.2

Table 3.5 Elasticsearch’s similarity algorithms (view table figure)

Similarity algorithm	Type	Description
Okapi BM25 (default)	`BM25`	An enhanced TF/IDF algorithm that considers field length in addition to term and document frequencies
Divergence from Randomness (DFR)	`DFR`	Uses the DFR framework developed by its authors, Amati and Rijsbergen, which aims to improve search relevance by measuring the divergence between the actual term and an expected random distribution. Terms that occur more often in relevant documents than in a random distribution are assigned higher weights when ranking search results.
Divergence from Independence (DFI)	`DFI`	A specific model of the DFR family that measures the divergence of the actual term frequency distribution from an independent distribution. DFI aims to assign higher scores to documents by comparing the observed term frequencies with those expected in a random, uncorrelated term frequency.
LM Dirichlet	`LMDirichlet`	Calculates the relevance of documents based on the probability of generating the query terms from the document’s language model
LM Jelinek-Mercer	`LMJelinek-Mercer`	Provides improved search result relevance compared to models that do not account for data sparsity
Manua		Creates a manual script
Boolean similarity	`boolean`	Does not consider ranking factors unless the query criteria are satisfied

Compare Elasticsearch’s built-in similarity choices when deciding whether BM25 fits a field’s relevance requirements. section 3.4.2

How Elasticsearch in Action, Second Edition covers it

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.