Lexical search retrieves documents by matching query words and ranking their occurrence and importance. It selectively extracts relevant portions when files or document collections are too large to place directly into context. section 5.1.3 section 5.2.1
Core mechanics and trade-offs:
- TF-IDF multiplies term frequency within a document by inverse document frequency across the collection, reducing the importance of common words. section 5.2.1
- BM25 extends this approach by adjusting for document length and the saturation effect of repeated terms. section 5.2.1
- Exact matching makes lexical search especially effective for function names, error codes, configuration values, special characters, and code syntax. section 5.2.1 section 5.2.2
- Lexical search does not recognize synonyms or semantically related expressions without separate processing, so hybrid search can combine it with vector search. section 5.2.1 section 5.2.2
Exact identifiers versus meaning
A search for get_user_by_id, 404, or MAX_CONNECTIONS benefits from lexical matching. A search for sofa will not reliably retrieve a document containing couch unless a synonym dictionary or another search method is added. section 5.2.1
Figure 5.4 Although sofa and couch have similar meanings, a keyword search can’t determine their similarity without separate processing. Common terms such as is that appear frequently aren’t considered important (shown with triangles). The left example is judged relevant because it overlaps with sofa, but the right example is judged irrelevant despite being semantically relevant.
Contrasts literal overlap with semantic relevance: common words receive low importance, while sofa and couch remain unrelated without separate processing. section 5.2.1
