liveBook · Manning

Lexical search

As explained in Building Reliable AI Systems

A retrieval method that searches using sparse representations of content, such as term-based vectors.

Lexical search retrieves content using sparse, term-based representations. In a retrieval-augmented generation (RAG) pipeline, it can complement dense embedding search as part of hybrid retrieval. The practical goal is to surface the right context for a user’s question: even an accurate document and a capable language model cannot compensate for a missed retrieval. section 3.8.12 section 4.0

Hybrid pipelines can combine dense and sparse retrieval with reranking to improve precision. The available treatment identifies that design but does not specify a lexical index, scoring method, or fusion algorithm. Chunking, retrieval quality, and continuous evaluation remain production concerns. section 4.0 section 3.8.12

Do not mistake keyword checks elsewhere in an application for a full lexical retriever. A blocked-word guardrail or keyword-based model router tests for specified strings; neither ranks documents for retrieval. Both illustrate a limitation of literal matching: synonyms, creative phrasing, or unexpectedly complex queries can defeat simple lists. section 6.4.2 section 9.2.4

How Building Reliable AI Systems covers it

How deep it goes
Identifies sparse retrieval as a hybrid-pipeline component but does not detail its indexing, scoring, or implementation.
Examples
Uses an insurance-policy chatbot to show the cost of missed context; keyword guardrails and routing illustrate separate literal-match limitations.
What you can do after
Evaluate/understand where a sparse component fits in a hybrid RAG pipeline without treating simple keyword checks as document retrieval.
How it connects
Connects to dense embeddings, hybrid retrieval, reranking, chunking, and retrieval evaluation.

Also explained in

Compare how 8 books explain lexical search →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Reliable AI Systems. The section links point to the parts of the book it draws on.