liveBook · Manning

Inverse document frequency

As explained in Build an Advanced RAG Application (From Scratch)

Inverse document frequency is a corpus-level measure that assigns lower values to terms appearing in many documents and higher values to rarer terms.

Inverse document frequency (IDF) measures how informative a term is across a document collection. It assigns lower values to terms appearing in many documents and higher values to rarer terms, helping keyword search distinguish useful matches from broadly shared words. section 3.2

IDF is combined with term frequency in TF-IDF. For a term, the implementation computes document frequency as the number of indexed documents containing it, then calculates IDF as log(total documents / (document frequency + 1)). Multiplying this value by the term's frequency in a document produces its TF-IDF weight. section 3.2.1

A basic keyword-ranking pipeline applies IDF through these steps:

  • Tokenize each document into words or other indexable terms. section 3.2
  • Build an inverted index mapping each term to the document identifiers where it appears. section 3.2
  • Count the documents containing a query term and calculate its smoothed IDF. section 3.2.1
  • Add each matching term's TF-IDF contribution to its document score, then sort documents by descending score. section 3.2.1

Ranking “machine learning”

In a six-sentence corpus, the query matches sentence 3 on both “machine” and “learning,” producing a score of 1.791759469228055. Sentence 4 matches only “learning” and scores 0.6931471805599453; documents without either query term score zero. section 3.2.1

Listing 3.1 Create tokens and inverted index
import nltk  #A
nltk.download('punkt')
from nltk.tokenize import word_tokenize
import math
 
examples= [
    "The cat is playing in the garden",      #B
    "A dog and a cat are good pets",
    "Cats love to chase mice", 
    "Machine learning is based on algorithms",
    "Deep learning uses neural networks", 
    "Recurrent networks have connections"
]
 
tokenized_docs = [word_tokenize(d) for d in examples]  #C
 
index = {} #D
 
for i, doc in enumerate(tokenized_docs):
    for word in doc:
        if word not in index:
            index[word] = [] 
        index[word].append(i)  
 
print(f'Inverted Index:\n {index}')

Inspect the runnable Python setup for tokenization and inverted-index construction before applying IDF. section 3.2.1

Listing 3.2 Create TF-IDF Function
def tfidf(word, id):   #A
 
    tf = tokenized_docs[id].count(word)  #B
 
    df = len(index[word]) if word in index else 0  #C
 
    idf =  math.log(len(examples) / (df+1))  #D
 
   return tf * idf

Inspect the compact TF-IDF function that derives document frequency, applies logarithmic IDF, and multiplies by term frequency. section 3.2.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines corpus weighting, gives a smoothed logarithmic formula, and implements ranking in Python without broader retrieval internals.
Examples
Uses a six-sentence corpus, a machine-learning query, inverted-index output, and ranked TF-IDF scores.
What it assumes
Assumes basic text collections and arithmetic while introducing tokenization, inverted indexes, document frequency, and TF-IDF incrementally.
What you can do after
Build it from scratch in Python by computing IDF, scoring matching documents, and sorting results.
How it connects
Connects TF-IDF with term frequency, inverted index, tokenization, and keyword search.

Also explained in

Compare how 6 books explain inverse document frequency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.