How deeply does Generative AI in Action cover lexical search?

Explains TF-IDF keyword matching with inverted indexes and contrasts sparse and dense retrieval by capability and cost.

Lexical search

A sparse retriever is an information-retrieval system that matches queries and documents through sparse lexical representations and keyword-based statistics.

Lexical search matches queries to documents through keyword overlap. In retrieval-augmented generation (RAG), a retriever searches knowledge sources and supplies relevant information to a generator. section 7.4

Sparse retrieval commonly uses TF-IDF and inverted indexes:

It is fast and efficient for large-scale, keyword-dependent search when computational resources are limited. section 7.4
It can struggle with synonyms and nuanced phrasing. Dense retrieval can capture semantic relationships, but is more computationally intensive and requires substantial training data. section 7.4

Choose between sparse and dense retrieval based on query characteristics, domain specificity, available resources, and the need to understand nuanced language. section 7.4

How Generative AI in Action covers it

liveBook manning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

AI-Powered SearchKeyword matching is a retrieval technique that identifies content by comparing query keywords with indexed textual terms.
liveBook manning.com

Starting Data Analytics with Generative AI and PythonA text-mining technique that locates and extracts information from text collections by matching specified words or phrases.
liveBook manning.com

The Art of AI Product DevelopmentLexical search retrieves and ranks documents by matching words or phrases in their textual forms.
liveBook manning.com

AI Agents and ApplicationsLexical search retrieves content primarily by matching literal words or terms.
liveBook manning.com

Building Reliable AI SystemsA retrieval method that searches using sparse representations of content, such as term-based vectors.
liveBook manning.com

Build an AI Agent (From Scratch)Keyword search is an information-retrieval technique that ranks documents according to the occurrence and importance of query words.
liveBook manning.com

AI Agents in Action, Second EditionKeyword matching retrieves content by comparing literal terms in a query with terms in stored content.

Compare how 8 books explain lexical search →

Related concepts

hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
similarity searchcontrasts with · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
BM25includes · BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
inverted indexuses · An inverted index is a data structure that maps terms or tokens to the documents, records, or locations in which they occur.
term frequency-inverse document frequencyuses · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
reciprocal rank fusionused by · Reciprocal rank fusion is a rank-aggregation technique that combines ranked lists by summing reciprocal-rank contributions for each item across the lists.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.