Lexical search retrieves document chunks by matching query words to words in the content. Unlike dense retrieval, which compares embeddings for semantic similarity, it focuses on word-level similarity and precise keyword matches. section 6.2.2 section 10.1
A sparse retrieval pipeline works by:
- Tokenizing each chunk and building an inverted index that maps tokens to the chunks containing them. section 10.1
- Tokenizing the query in the same way, looking up its terms, then ranking matching chunks with methods such as BM25 or TF-IDF. section 10.1
- Using keyword matches for precise queries, Boolean constraints such as “must” and “must not,” and results whose matches are easy to explain. section 10.1
Lexical and semantic retrieval can be combined in hybrid search, with Reciprocal Rank Fusion merging their ranked results. BM25 depends on inverted-index data structures; LangChain’s BM25Retriever works on in-memory document lists but does not scale to millions of documents. section 10.8 section 10.8
