Lexical search
Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
AI-Powered Search
Keyword matching is a retrieval technique that identifies content by comparing query keywords with indexed textual terms.
- How it is used
- Provides a default matching layer and a baseline for improving search relevance.
- How deep it goes
- Explains inverted-index lookups and conceptual term vectors, demonstrates search requests, and defers detailed scoring calculations.
- Examples
- Uses product searches for “ipod” and restaurant-review searches for “bbq near charlotte.”
- What you can do after
- Evaluate and understand keyword-matching results and their relevance limitations.
Starting Data Analytics with Generative AI and Python
A text-mining technique that locates and extracts information from text collections by matching specified words or phrases.
- How it is used
- Locates customer feedback on specific features and uses keyword matches as a starting point for sentiment classification.
- How deep it goes
- Works through keyword matching, cleaning, and classification evaluation, then compares the baseline with advanced sentiment methods.
- Examples
- Uses Portuguese customer reviews to extract positive feedback and compares results against review scores.
- What you can do after
- Apply it via Python and pandas to filter reviews, then evaluate the classifications against a reference.
Read how Starting Data Analytics with Generative AI and Python explains lexical search →
Retrieval Augmented Generation, The Foundational Ideas
Lexical search retrieves text by matching words or other surface forms rather than representing meaning directly.
- How it is used
- Narrows a large corpus to candidate passages and handles queries where exact terms, entities, or codes matter.
- How deep it goes
- Explains BM25 through a scored comparison and gives a vocabulary-coverage selection procedure, without detailing BM25’s ranking formula.
- Examples
- Compares car-related document rankings and contrasts a 401(k) question with a differently worded policy document.
- What you can do after
- Evaluate/understand it by measuring query-token coverage and comparing exact-match needs with semantic retrieval needs.
Read how Retrieval Augmented Generation, The Foundational Ideas explains lexical search →
Build an Advanced RAG Application (From Scratch)
Keyword search retrieves documents or items by matching query words or phrases against words or phrases in stored content.
- How it is used
- Provides a scalable information-retrieval baseline for finding documents through query-term overlap.
- How deep it goes
- Builds tokenization, inverted indexing, TF-IDF scoring, and ranked retrieval step by step in Python.
- Examples
- Ranks six tokenized example sentences for the query “machine learning” using summed TF-IDF scores.
- What you can do after
- Build it from scratch with an inverted index and TF-IDF ranking for a small text corpus.
Read how Build an Advanced RAG Application (From Scratch) explains lexical search →
The Art of AI Product Development
Lexical search retrieves and ranks documents by matching words or phrases in their textual forms.
- How it is used
- Improves retrieval precision for technical, company-specific, taxonomic, and ambiguous queries.
- How deep it goes
- Explains exact matching, BM25 ranking, and combination with semantic search, but stops short of implementation details.
- Examples
- Uses augmented workflow, API endpoints, and airline business-class queries to demonstrate exact matching and ambiguity.
- What you can do after
- Apply it via BM25 ranking and combine it with semantic search to improve retrieval precision.
Read how The Art of AI Product Development explains lexical search →
AI Agents and Applications
Lexical search retrieves content primarily by matching literal words or terms.
- How it is used
- Retrieves document chunks for precise, keyword-driven queries in LLM applications.
- How deep it goes
- Explains tokenization, inverted-index lookup, and BM25 or TF-IDF ranking, without detailing ranking algorithms.
- Examples
- Illustrates Boolean constraints with “must” and “must not” and contrasts keyword matching with semantic retrieval.
- What you can do after
- Understand how matching query terms to indexed chunks enables precise retrieval and explainable results.
Read how AI Agents and Applications explains lexical search →
Designing AI Systems
Keyword search retrieves content by matching literal terms or tokens in a query against indexed content.
- How it is used
- Supports exact retrieval of identifiers, citations, phrases, and other terms that semantic similarity may dilute.
- How deep it goes
- Defines strengths, limitations, interface behavior, and hybrid orchestration, but stops short of backend-specific matching implementation.
- Examples
- Uses ERR-4012 and HIPAA 164.512(a) to demonstrate exact identifier and citation retrieval.
- What you can do after
- Build keyword retrieval through a VectorStore interface and combine it with vector results through hybrid search.
Building Reliable AI Systems
A retrieval method that searches using sparse representations of content, such as term-based vectors.
- How it is used
- Combines sparse and dense retrieval in RAG pipelines to improve document selection.
- How deep it goes
- Introduces hybrid retrieval and illustrates simple keyword rules, but does not explain sparse retrieval scoring or implementation.
- Examples
- Uses policy chatbots, a travel-assistant content filter, and keyword-based model routing as examples.
- What you can do after
- Evaluate term-based approaches against semantic alternatives and recognize limitations of simple keyword rules.
Read how Building Reliable AI Systems explains lexical search →
LLM Evaluation and Alignment, The Foundational Ideas
Sparse retrieval ranks documents using statistical weights for query terms, typically with sparse keyword representations.
- How it is used
- Ranks indexed documents for information retrieval using statistically weighted query terms.
- How deep it goes
- Explains term-frequency and inverse-document-frequency scoring, BM25 saturation, document-length normalization, and corpus indexing.
- Examples
- Uses medical queries involving “myocardial infarction,” “heart attack,” and “MI” to illustrate terminology mismatch.
- What you can do after
- Evaluate retrieval with precision@k, recall@k, and AP/MAP while checking for paraphrase failures.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains lexical search →
Build an AI Agent (From Scratch)
Keyword search is an information-retrieval technique that ranks documents according to the occurrence and importance of query words.
- How it is used
- Supports retrieval of relevant portions from large files and document collections, especially when exact terms matter.
- How deep it goes
- Defines TF-IDF and BM25, explains ranking behavior and limitations, but does not detail implementation.
- Examples
- Uses get_user_by_id, 404, MAX_CONNECTIONS, and the sofa-couch contrast to demonstrate exact and failed matches.
- What you can do after
- Evaluate when exact-term retrieval should supplement semantic search in a production retrieval system.
Read how Build an AI Agent (From Scratch) explains lexical search →
Enterprise RAG
Keyword search retrieves records by matching query terms against indexed text or metadata.
- How it is used
- Supports enterprise retrieval by combining literal term matching with semantic search before ranking results for answer generation.
- How deep it goes
- Implements a hybrid query and ranking workflow but stops short of explaining text-index internals or ranking algorithms.
- Examples
- Uses “Dubious Parenting Advice,” “X24 blender warranty,” and the returned “Dubious Parenting Tips” record as concrete search examples.
- What you can do after
- Apply it via Azure AI Search to submit text and vector queries, retrieve records, and combine their results.
Generative AI in Action
A sparse retriever is an information-retrieval system that matches queries and documents through sparse lexical representations and keyword-based statistics.
- How it is used
- Searches knowledge corpora for relevant information to supply a RAG generator.
- How deep it goes
- Explains TF-IDF keyword matching with inverted indexes and contrasts sparse and dense retrieval by capability and cost.
- Examples
- Uses TF-IDF and keyword overlap to illustrate sparse retrieval.
- What you can do after
- Evaluate sparse versus dense retrieval against query needs, domain specificity, and available resources.
Agent Design Patterns
Keyword-based search retrieves information by matching query terms against terms in the searchable data.
- How it is used
- Uses keyword matching to retrieve documents, especially when queries contain exact identifiers or rare terms.
- How deep it goes
- Defines sparse retrieval as a weighted keyword vector and explains its strengths and limits, without detailing indexing or scoring internals.
- Examples
- Uses technical-support searches for error codes and product identifiers alongside vague symptom descriptions.
- What you can do after
- Apply via BM25 as a sparse search component, combining its results with dense retrieval and re-ranking for technical-support queries.
AI Agents in Action, Second Edition
Keyword matching retrieves content by comparing literal terms in a query with terms in stored content.
- How it is used
- Retrieves information in RAG by matching literal query terms against stored content.
- How deep it goes
- Defines lexical matching by contrasting it with semantic similarity, without detailing its implementation.
- Examples
- Uses an exact-time question about a lightning strike to demonstrate hybrid keyword and vector retrieval.
- What you can do after
- Apply keyword search alongside vector search in a hybrid RAG agent and inspect the retrieval calls.
Read how AI Agents in Action, Second Edition explains lexical search →
Also covered in
- Context EngineeringIn early access (MEAP)
- Effective Conversational AI
- Designing AI AgentsIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- vector databaseA vector database is a data-management system that stores and indexes numerical vector representations and supports retrieval through similarity or distance searches.
- BM25BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
- similarity searchSimilarity search is a retrieval technique that finds or ranks items by their similarity to a query, often by comparing feature or vector representations under a similarity or distance measure.
Related concepts
- hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
- similarity searchalternative to · Similarity search is a retrieval technique that finds or ranks items by their similarity to a query, often by comparing feature or vector representations under a similarity or distance measure.
- BM25includes · BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
- inverted indexuses · An inverted index is a data structure that maps terms or tokens to the documents, records, or locations in which they occur.
- term frequency-inverse document frequencyuses · Term frequency-inverse document frequency (TF-IDF) is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
Questions about lexical search
What is lexical search?
Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
Which Manning books explain lexical search?
AI-Powered Search, Starting Data Analytics with Generative AI and Python, Retrieval Augmented Generation, The Foundational Ideas, Build an Advanced RAG Application (From Scratch), The Art of AI Product Development, AI Agents and Applications, Designing AI Systems, Building Reliable AI Systems and 6 more.
What should I know before learning lexical search?
Manning books that teach lexical search usually introduce language model, vector database, BM25 and similarity search first.
What is the difference between lexical search and similarity search?
Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting. Similarity search is a retrieval technique that finds or ranks items by their similarity to a query, often by comparing feature or vector representations under a similarity or distance measure.
What kinds of lexical search are there?
Kinds of lexical search include BM25.
What is lexical search also called?
Lexical search is also called keyword search, keyword matching, sparse retrieval, keyword-based search or sparse search.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.













