An inverted index supports lexical search by mapping each unique search token to the document chunks where it appears. During indexing, chunks are tokenized; during querying, the question is tokenized the same way and matched against those postings. Search tokenization is optimized for retrieval, not language modeling. section 10.1
A practical sparse-retrieval workflow is:
- Tokenize each document chunk with a search-oriented tokenizer. section 10.1
- Create postings that associate every unique token with the chunks containing it. section 10.1
- Tokenize the query identically and retrieve chunks matching its terms. section 10.1
- Rank matches with BM25 or TF–IDF, then combine them with semantic results when needed. section 10.1 section 10.8
Keyword-driven retrieval
A question containing a distinctive term retrieves chunks whose postings contain that term. Sparse search can also enforce Boolean conditions such as “must” and “must not,” making matches precise and explainable because results link directly to query terms. section 10.1
The structure favors exact lexical matches and transparent relevance explanations, while dense retrieval captures semantic similarity through embeddings. BM25 may require index support that many vector stores lack; LangChain’s BM25Retriever operates on in-memory document lists and does not scale to millions of documents. section 10.1 section 10.8
