Lexical search retrieves documents by overlapping query terms or phrases with stored text. It is a common information-retrieval baseline: fast and scalable, but centered on word frequency and exact surface forms rather than contextual meaning. section 3.2 section 3.5
An inverted index makes matching efficient by mapping each token to the document IDs that contain it. A query can therefore look up its terms directly instead of scanning every document. Building the index starts by tokenizing each document, then adding its ID to the posting list for every token. section 3.2 section 3.2.1
Listing 3.1 Create tokens and inverted index
import nltk #A
nltk.download('punkt')
from nltk.tokenize import word_tokenize
import math
examples= [
"The cat is playing in the garden", #B
"A dog and a cat are good pets",
"Cats love to chase mice",
"Machine learning is based on algorithms",
"Deep learning uses neural networks",
"Recurrent networks have connections"
]
tokenized_docs = [word_tokenize(d) for d in examples] #C
index = {} #D
for i, doc in enumerate(tokenized_docs):
for word in doc:
if word not in index:
index[word] = []
index[word].append(i)
print(f'Inverted Index:\n {index}')
A compact Python implementation tokenizes six sentences and constructs a dictionary from each word to the IDs of sentences containing it. section 3.2.1
TF-IDF ranks the matched documents. Term frequency counts a query word within a document; inverse document frequency downweights words that occur across many documents. Sum each query term’s TF-IDF weight for every matching document, then sort scores descending. section 3.2.1
Ranking “machine learning”
Against a six-sentence corpus, the query “machine learning” ranks “Machine learning is based on algorithms” above “Deep learning uses neural networks”: the first matches both query terms, while the second matches only “learning.” section 3.2.1
Lexical search is useful when exact wording matters, but its retrieval behavior has clear limits:
- It focuses on short individual terms rather than the query’s holistic meaning. section 3.2
- It can miss relevant content that does not repeat the query’s words or phrases. section 3.5
- Semantic search instead encodes queries and documents as vectors and compares their meaning with cosine similarity. section 3.5













