Lexical search
Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
AI-Powered Search
Keyword matching is a retrieval technique that identifies content by comparing query keywords with indexed textual terms.
- How it is used
- Provides an explicit-query matching baseline and a fallback when richer query interpretation cannot resolve a query.
- How deep it goes
- Explains inverted-index postings, sparse term vectors, and ranking at a high level; defers the specific scoring calculation.
- Examples
- Tests product-catalog searches, restaurant-review queries, and outdoor-question queries to expose useful matches and misleading keyword overlap.
- What you can do after
- Apply it via an indexed collection and search request, then evaluate result relevance before adding richer matching.
Starting Data Analytics with Generative AI and Python
A text-mining technique that locates and extracts information from text collections by matching specified words or phrases.
- How it is used
- Locates reviews about specified features or opinions when broad frequency counts do not answer a business question.
- How deep it goes
- Works through generated filtering code, cleaning choices, and classification checks, while showing that default AI code may omit needed handling.
- Examples
- Uses Portuguese customer-review filtering and a 500-review sentiment comparison to expose the strengths and misses of keyword matching.
- What you can do after
- Apply it via pandas to extract matching reviews, then evaluate positive-review labels with sensitivity and specificity.
Read how Starting Data Analytics with Generative AI and Python explains lexical search →
The Art of AI Product Development
Lexical search retrieves and ranks documents by matching words or phrases in their textual forms.
- How it is used
- Improves retrieval precision for technical, company-specific, taxonomic, and ambiguous queries.
- How deep it goes
- Explains exact matching, BM25 ranking, and combination with semantic search, but stops short of implementation details.
- Examples
- Uses augmented workflow, API endpoints, and airline business-class queries to demonstrate exact matching and ambiguity.
- What you can do after
- Apply it via BM25 ranking and combine it with semantic search to improve retrieval precision.
Read how The Art of AI Product Development explains lexical search →
AI Agents and Applications
Lexical search retrieves content primarily by matching literal words or terms.
- How it is used
- Retrieves document chunks for precise, keyword-driven queries in LLM applications.
- How deep it goes
- Explains tokenization, inverted-index lookup, and BM25 or TF-IDF ranking, without detailing ranking algorithms.
- Examples
- Illustrates Boolean constraints with “must” and “must not” and contrasts keyword matching with semantic retrieval.
- What you can do after
- Understand how matching query terms to indexed chunks enables precise retrieval and explainable results.
Read how AI Agents and Applications explains lexical search →
Building Reliable AI Systems
A retrieval method that searches using sparse representations of content, such as term-based vectors.
- How it is used
- Complements dense embedding search in hybrid RAG retrieval intended to surface relevant context.
- How deep it goes
- Identifies sparse retrieval as a hybrid-pipeline component but does not detail its indexing, scoring, or implementation.
- Examples
- Uses an insurance-policy chatbot to show the cost of missed context; keyword guardrails and routing illustrate separate literal-match limitations.
- What you can do after
- Evaluate/understand where a sparse component fits in a hybrid RAG pipeline without treating simple keyword checks as document retrieval.
Read how Building Reliable AI Systems explains lexical search →
Build an AI Agent (From Scratch)
Keyword search is an information-retrieval technique that ranks documents according to the occurrence and importance of query words.
- How it is used
- Supports retrieval of relevant portions from large files and document collections, especially when exact terms matter.
- How deep it goes
- Defines TF-IDF and BM25, explains ranking behavior and limitations, but does not detail implementation.
- Examples
- Uses get_user_by_id, 404, MAX_CONNECTIONS, and the sofa-couch contrast to demonstrate exact and failed matches.
- What you can do after
- Evaluate when exact-term retrieval should supplement semantic search in a production retrieval system.
Read how Build an AI Agent (From Scratch) explains lexical search →
Generative AI in Action
A sparse retriever is an information-retrieval system that matches queries and documents through sparse lexical representations and keyword-based statistics.
- How it is used
- Searches knowledge corpora for relevant information to supply a RAG generator.
- How deep it goes
- Explains TF-IDF keyword matching with inverted indexes and contrasts sparse and dense retrieval by capability and cost.
- Examples
- Uses TF-IDF and keyword overlap to illustrate sparse retrieval.
- What you can do after
- Evaluate sparse versus dense retrieval against query needs, domain specificity, and available resources.
AI Agents in Action, Second Edition
Keyword matching retrieves content by comparing literal terms in a query with terms in stored content.
- How it is used
- Retrieves information in RAG by matching literal query terms against stored content.
- How deep it goes
- Defines lexical matching by contrasting it with semantic similarity, without detailing its implementation.
- Examples
- Uses an exact-time question about a lightning strike to demonstrate hybrid keyword and vector retrieval.
- What you can do after
- Apply keyword search alongside vector search in a hybrid RAG agent and inspect the retrieval calls.
Read how AI Agents in Action, Second Edition explains lexical search →
Related concepts
- hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
- similarity searchcontrasts with · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
- BM25includes · BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
- inverted indexuses · An inverted index is a data structure that maps terms or tokens to the documents, records, or locations in which they occur.
- term frequency-inverse document frequencyuses · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
- reciprocal rank fusionused by · Reciprocal rank fusion is a rank-aggregation technique that combines ranked lists by summing reciprocal-rank contributions for each item across the lists.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.