liveBook · Manning

Lexical search

As explained in Retrieval Augmented Generation, The Foundational Ideas

Lexical search retrieves text by matching words or other surface forms rather than representing meaning directly.

A search for “automobile maintenance” can miss a relevant passage about a “car” when the passage shares none of the query’s words. Lexical search ranks text through surface-form matches, not directly through meaning. BM25 is a common keyword-ranking method that gives more weight to rare terms; without matching terms, a passage may receive a zero score. section 2.1.3

In open-domain question answering, a lexical retriever can rapidly narrow millions of documents to a small set of passages for a reader to examine. TF-IDF and BM25 are sparse retrieval methods: their word-based representations have many zero values. Fast candidate selection does not guarantee that the right passage survives, especially when a question and its answer use different vocabulary. section 3.1

When exact terms matter—and when they do not

A query containing an entity name or regulatory code can benefit from exact matching. A question about how much an employee can “put into” a 401(k), however, may miss a document phrased in terms of “maximum employee deferral.” A sparse searcher can refine exact-match queries with Lucene operators that add related terms, filter noise, or boost tokens; a dense searcher is better suited to broader thematic queries. section 3.1 section 11.3.2

--- BM25 (Keyword Search) Results ---
Doc 1: Score 1.17 | Text: 'The automobile industry is booming'
Doc 2: Score 0.36 | Text: 'Vehicle maintenance can prevent breakdowns'
Doc 4: Score 0.33 | Text: 'Automobiles and cars require regular maintenance'
Doc 0: Score 0.00 | Text: 'I bought a new car today'
Doc 3: Score 0.00 | Text: 'She repaired the computer after the crash'
 
--- Semantic Search Results ---
Doc 4: Score 0.79 | Text: 'Automobiles and cars require regular maintenance'
Doc 2: Score 0.75 | Text: 'Vehicle maintenance can prevent breakdowns'
Doc 1: Score 0.69 | Text: 'The automobile industry is booming'
Doc 3: Score 0.55 | Text: 'She repaired the computer after the crash'
Doc 0: Score 0.54 | Text: 'I bought a new car today'

The five-document comparison makes the tradeoff visible: BM25 gives the car-only document a score of 0.00 for “automobile maintenance,” while semantic search makes it retrievable but still ranks an unrelated computer document slightly higher. section 2.1.3

Choose a retrieval approach by testing the expected queries against the corpus:

  • Collect representative queries, normalize query and document tokens, and measure the share of query tokens present in the document vocabulary. section 2.1.6
  • With high overlap—suggested here as above 70%—start with BM25, particularly for precise legal or technical terms. With low overlap—below 30%—consider semantic retrieval to address vocabulary mismatch. section 2.1.6
  • For mixed needs, route exact entities to sparse search and thematic questions to dense search. Inspect broad or irrelevant results and refine the query or change retrieval strategy. section 11.3.2 section 11.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains BM25 through a scored comparison and gives a vocabulary-coverage selection procedure, without detailing BM25’s ranking formula.
Examples
Compares car-related document rankings and contrasts a 401(k) question with a differently worded policy document.
What it assumes
Introduces sparse word-count representations while using tokenized queries and a short Python ranking comparison.
What you can do after
Evaluate/understand it by measuring query-token coverage and comparing exact-match needs with semantic retrieval needs.
How it connects
Connects lexical search to sparse retrieval, TF-IDF, BM25, semantic retrieval, vocabulary mismatch, and question answering.

Also explained in

Compare how 14 books explain lexical search →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.