Inverse document frequency (IDF) measures how informative a term is across a document collection. It assigns lower values to terms appearing in many documents and higher values to rarer terms, helping keyword search distinguish useful matches from broadly shared words. section 3.2
IDF is combined with term frequency in TF-IDF. For a term, the implementation computes document frequency as the number of indexed documents containing it, then calculates IDF as log(total documents / (document frequency + 1)). Multiplying this value by the term's frequency in a document produces its TF-IDF weight. section 3.2.1
A basic keyword-ranking pipeline applies IDF through these steps:
- Tokenize each document into words or other indexable terms. section 3.2
- Build an inverted index mapping each term to the document identifiers where it appears. section 3.2
- Count the documents containing a query term and calculate its smoothed IDF. section 3.2.1
- Add each matching term's TF-IDF contribution to its document score, then sort documents by descending score. section 3.2.1
Ranking “machine learning”
In a six-sentence corpus, the query matches sentence 3 on both “machine” and “learning,” producing a score of 1.791759469228055. Sentence 4 matches only “learning” and scores 0.6931471805599453; documents without either query term score zero. section 3.2.1
Listing 3.1 Create tokens and inverted index
import nltk #A
nltk.download('punkt')
from nltk.tokenize import word_tokenize
import math
examples= [
"The cat is playing in the garden", #B
"A dog and a cat are good pets",
"Cats love to chase mice",
"Machine learning is based on algorithms",
"Deep learning uses neural networks",
"Recurrent networks have connections"
]
tokenized_docs = [word_tokenize(d) for d in examples] #C
index = {} #D
for i, doc in enumerate(tokenized_docs):
for word in doc:
if word not in index:
index[word] = []
index[word].append(i)
print(f'Inverted Index:\n {index}')
Inspect the runnable Python setup for tokenization and inverted-index construction before applying IDF. section 3.2.1
Listing 3.2 Create TF-IDF Function
def tfidf(word, id): #A
tf = tokenized_docs[id].count(word) #B
df = len(index[word]) if word in index else 0 #C
idf = math.log(len(examples) / (df+1)) #D
return tf * idf
Inspect the compact TF-IDF function that derives document frequency, applies logarithmic IDF, and multiplies by term frequency. section 3.2.1





