liveBook · Manning

Term frequency-inverse document frequency

As explained in Build an Advanced RAG Application (From Scratch)

TF-IDF is a document-term weighting and scoring scheme that combines a term's within-document frequency with its inverse frequency across a document collection.

TF-IDF weights a term by combining its frequency within one document with its frequency across a document collection. It supports keyword search by scoring documents that overlap with query terms, while an inverted index makes matching documents quick to find. section 3.2

A small keyword-search implementation follows this pipeline:

  • Tokenize each document into words using NLTK's word_tokenize function. section 3.2.1
  • Build an inverted index that maps each word to the document indices containing it. section 3.2

Ranking a machine-learning query

For the query "machine learning," the implementation scores "Machine learning is based on algorithms" highest at 1.791759469228055. "Deep learning uses neural networks" follows at 0.6931471805599453, while documents without matching query terms receive zero.

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Calculates TF-IDF and query rankings in Python, but stops short of broader production-scale search trade-offs.
Examples
Uses six toy sentences, an inverted-index listing, and a machine-learning query with printed ranked scores.
What it assumes
Assumes basic Python while teaching tokenization, inverted-index construction, term frequency, and document frequency step by step.
What you can do after
Build it from scratch by tokenizing documents, indexing terms, calculating weights, and ranking query matches.
How it connects
Connects keyword search with tokenization, inverted indexes, document frequency, and document retrieval.

Also explained in

Compare how 12 books explain term frequency-inverse document frequency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.