liveBook · Manning

Lexical search

As explained in Build an Advanced RAG Application (From Scratch)

Keyword search retrieves documents or items by matching query words or phrases against words or phrases in stored content.

Lexical search retrieves documents by overlapping query terms or phrases with stored text. It is a common information-retrieval baseline: fast and scalable, but centered on word frequency and exact surface forms rather than contextual meaning. section 3.2 section 3.5

An inverted index makes matching efficient by mapping each token to the document IDs that contain it. A query can therefore look up its terms directly instead of scanning every document. Building the index starts by tokenizing each document, then adding its ID to the posting list for every token. section 3.2 section 3.2.1

Listing 3.1 Create tokens and inverted index
import nltk  #A
nltk.download('punkt')
from nltk.tokenize import word_tokenize
import math
 
examples= [
    "The cat is playing in the garden",      #B
    "A dog and a cat are good pets",
    "Cats love to chase mice", 
    "Machine learning is based on algorithms",
    "Deep learning uses neural networks", 
    "Recurrent networks have connections"
]
 
tokenized_docs = [word_tokenize(d) for d in examples]  #C
 
index = {} #D
 
for i, doc in enumerate(tokenized_docs):
    for word in doc:
        if word not in index:
            index[word] = [] 
        index[word].append(i)  
 
print(f'Inverted Index:\n {index}')

A compact Python implementation tokenizes six sentences and constructs a dictionary from each word to the IDs of sentences containing it. section 3.2.1

TF-IDF ranks the matched documents. Term frequency counts a query word within a document; inverse document frequency downweights words that occur across many documents. Sum each query term’s TF-IDF weight for every matching document, then sort scores descending. section 3.2.1

Ranking “machine learning”

Against a six-sentence corpus, the query “machine learning” ranks “Machine learning is based on algorithms” above “Deep learning uses neural networks”: the first matches both query terms, while the second matches only “learning.” section 3.2.1

Lexical search is useful when exact wording matters, but its retrieval behavior has clear limits:

  • It focuses on short individual terms rather than the query’s holistic meaning. section 3.2
  • It can miss relevant content that does not repeat the query’s words or phrases. section 3.5
  • Semantic search instead encodes queries and documents as vectors and compares their meaning with cosine similarity. section 3.5

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Builds tokenization, inverted indexing, TF-IDF scoring, and ranked retrieval step by step in Python.
Examples
Ranks six tokenized example sentences for the query “machine learning” using summed TF-IDF scores.
What it assumes
Introduces tokenization and inverted indexing from first principles before implementing search.
What you can do after
Build it from scratch with an inverted index and TF-IDF ranking for a small text corpus.
How it connects
Contrasts keyword search with semantic search, Sentence Transformers, vector representations, and cosine similarity.

Also explained in

Compare how 14 books explain lexical search →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.