Term frequency-inverse document frequency
Term frequency-inverse document frequency (TF-IDF) is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
Natural Language Processing in Action, Second Edition
A TF-IDF model represents text using term-frequency and inverse-document-frequency weights for terms in documents.
- How it is used
- Ranks documents and supplies baseline text features for search, spam filtering, classification, and semantic-search pipelines.
- How deep it goes
- Derives logarithmic weighting, smoothing, sparse-matrix ranking, and cosine comparison, but defers semantic meaning representation.
- Examples
- Uses a Hacker Public Radio corpus, a lost-audio query, sparse matrices, and logarithmic TF-IDF code.
- What you can do after
- Apply it via scikit-learn to vectorize documents, preserve sparsity, compare vectors, and rank search results.
AI Agents in Action, Second Edition
TF-IDF is a term-weighting technique that scores a word by its frequency in a document relative to its frequency across a collection of documents.
- How it is used
- Provides a keyword-based baseline for document similarity and retrieval.
- How deep it goes
- Defines term weighting and demonstrates vectorization and cosine similarity, then contrasts the approach with semantic embeddings.
- Examples
- Uses sentences about blue skies and breakfast, plus exercises comparing “blue” and “breakfast” similarity results with embedding-based rankings.
- What you can do after
- Apply it via scikit-learn’s TfidfVectorizer and cosine similarity to compare documents.
Read how AI Agents in Action, Second Edition explains term frequency-inverse document frequency →
Build an Advanced RAG Application (From Scratch)
TF-IDF is a document-term weighting and scoring scheme that combines a term's within-document frequency with its inverse frequency across a document collection.
- How it is used
- Ranks documents for short keyword queries within a retrieval pipeline.
- How deep it goes
- Calculates TF-IDF and query rankings in Python, but stops short of broader production-scale search trade-offs.
- Examples
- Uses six toy sentences, an inverted-index listing, and a machine-learning query with printed ranked scores.
- What you can do after
- Build it from scratch by tokenizing documents, indexing terms, calculating weights, and ranking query matches.
Elasticsearch in Action, Second Edition
A term frequency–inverse document frequency algorithm ranks documents using query-term frequency in documents and term rarity across the collection.
- How it is used
- Ranks full-text search results by weighting query-term frequency and collection-wide term rarity.
- How deep it goes
- Defines term and document frequency, demonstrates their relevance relationship, and stops short of detailed mathematical derivation.
- Examples
- Uses Java book titles, term-frequency counts, document-frequency effects, and comparisons between title and synopsis fields.
- What you can do after
- Evaluate why documents receive different relevance scores and when BM25 may better suit Elasticsearch search results.
AI-Powered Search
TF-IDF is a text-weighting metric that combines term frequency and inverse document frequency to estimate a term's importance within a document collection.
- How it is used
- Supports text-based relevance ranking by weighting query and document terms before similarity scoring.
- How deep it goes
- Defines TF and IDF, works through vector scoring, and stops short of detailed derivation or production tuning.
- Examples
- Uses the query “the cat in the hat,” three documents, Python vectors, dot products, and comparative relevance scores.
- What you can do after
- Apply TF-IDF by constructing query and document vectors and calculating similarity-based relevance scores.
Read how AI-Powered Search explains term frequency-inverse document frequency →
Math and Architectures of Deep Learning
A document-representation weighting scheme that combines term frequency with inverse document frequency to emphasize discriminative words.
- How it is used
- Supports document matching and retrieval by representing documents with discriminatively weighted term vectors.
- How deep it goes
- Defines TF, IDF, vector construction, and sparse storage, then connects the representation to cosine similarity and PCA-based topic modeling.
- Examples
- Uses “the” as a common-term example and contrasts gun counts across documents d0, d1, d3, and d6.
- What you can do after
- Apply TF-IDF with cosine similarity to match and retrieve documents.
Data Without Labels
Term frequency-inverse document frequency (TF-IDF) assigns a term a weight based on its frequency in a document and its inverse frequency across a document collection.
- How it is used
- Represents text for document comparison, clustering, categorization, and other language-data analyses.
- How deep it goes
- Explains TF and IDF definitions, gives a worked calculation, and demonstrates Python vectorization before clustering.
- Examples
- Uses medical terms, sports vocabulary, and a football-versus-travel Python clustering dataset.
- What you can do after
- Apply it via scikit-learn's TfidfVectorizer and use the resulting vectors in K-means text clustering.
Read how Data Without Labels explains term frequency-inverse document frequency →
AI Agents in Action
TF-IDF is a term-weighting method that scores words by their frequency in a document relative to their frequency across a document collection.
- How it is used
- Transforms documents into numerical vectors for similarity matching.
- How deep it goes
- Defines term frequency and inverse document frequency and works through a blue-word calculation in an eight-document corpus.
- Examples
- Uses “The sky is blue and beautiful,” an eight-document collection, and a scikit-learn vectorization example.
- What you can do after
- Apply TF–IDF with scikit-learn’s TfidfVectorizer to transform a document collection into vectors.
Read how AI Agents in Action explains term frequency-inverse document frequency →
Machine Learning System Design
A text-weighting representation that scores terms by their frequency in a document relative to their frequency across documents.
- How it is used
- Transforms sequential text data into features suitable for machine-learning algorithms.
- How deep it goes
- Defines TF-IDF as a text representation and notes its word-order limitation, but does not derive its weighting formula.
- Examples
- Uses text-to-feature transformation as the concrete example and contrasts it with longer N-grams and BERT embeddings.
- What you can do after
- Evaluate text representation choices for ML pipelines while accounting for lost word-order information.
Read how Machine Learning System Design explains term frequency-inverse document frequency →
LLM Evaluation and Alignment, The Foundational Ideas
TF-IDF is a lexical retrieval weighting scheme that combines a term’s frequency in a document with its inverse document frequency across a corpus.
- How it is used
- Deploys TF-IDF as a sparse retrieval method for terminology-sensitive document search in RAG systems.
- How deep it goes
- Explains term weighting, indexing, strengths, metrics, and paraphrase failure, while deferring formula-level implementation details.
- Examples
- Uses medical terminology such as “myocardial infarction,” “MI,” and “heart attack” to demonstrate matching and paraphrase gaps.
- What you can do after
- Evaluate sparse retrieval with precision@k, recall@k, and AP/MAP, then diagnose recall gaps caused by vocabulary mismatch.
Generative AI in Action
TF-IDF is a statistical weighting scheme that scores a term according to its frequency in a document and its rarity across a document collection.
- How it is used
- Retrieves relevant corpus information for RAG by matching query and document keywords.
- How deep it goes
- Explains weighting, inverted-index retrieval, and sparse-versus-dense trade-offs without mathematical derivation or implementation details.
- Examples
- Uses large-scale keyword-dependent search and RAG corpus retrieval as application contexts.
- What you can do after
- Evaluate/understand when sparse TF-IDF retrieval fits query language, domain needs, and available computational resources.
Read how Generative AI in Action explains term frequency-inverse document frequency →
Machine Learning Algorithms in Depth
TF-IDF is a term-weighting technique that combines a term's frequency in a document with the inverse frequency of documents containing that term.
- How it is used
- Adjusts word counts before topic modeling, emphasizing terms that discriminate documents.
- How deep it goes
- Defines the weighting formula and its variables, but does not derive or implement the method step by step.
- Examples
- Uses document corpora, vocabulary-count matrices, unigrams, and n-grams as representation examples.
- What you can do after
- Apply TF-IDF to reduce common-word influence and identify terms that distinguish documents.
Read how Machine Learning Algorithms in Depth explains term frequency-inverse document frequency →
Also covered in
- LLM Customization and Fine-TuningIn early access (MEAP)
- Ensemble Methods for Machine Learning
- AI Recommender SystemsIn early access (MEAP)
- Retrieval Augmented Generation, The Foundational IdeasIn early access (MEAP)
- Causal AI
- Build an AI Agent (From Scratch)
- Context EngineeringIn early access (MEAP)
What to know first
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- cosine similarityCosine similarity measures the directional alignment of two nonzero vectors as the cosine of the angle between them, equal to their dot product divided by the product of their magnitudes.
- BM25BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
Related concepts
- inverse document frequencyhas part · Inverse document frequency is a corpus-based measure of term rarity that assigns lower weights to terms appearing in many documents and higher weights to rarer terms.
- term frequencyhas part · Term frequency is the raw or normalized count of a term’s occurrences within a document or other text unit.
- lexical searchused by · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
- embeddingcontrasts with · An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- BM25alternative to · BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
- bag-of-words modelcontrasts with · A bag-of-words model represents text using word or token occurrences, counts, or weights while disregarding their order and syntactic relationships.
Questions about term frequency-inverse document frequency
What is term frequency-inverse document frequency?
Term frequency-inverse document frequency (TF-IDF) is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
Which Manning books explain term frequency-inverse document frequency?
Natural Language Processing in Action, Second Edition, AI Agents in Action, Second Edition, Build an Advanced RAG Application (From Scratch), Elasticsearch in Action, Second Edition, AI-Powered Search, Math and Architectures of Deep Learning, Data Without Labels, AI Agents in Action and 4 more.
What should I know before learning term frequency-inverse document frequency?
Manning books that teach term frequency-inverse document frequency usually introduce embedding, retrieval-augmented generation, cosine similarity and BM25 first.
What is the difference between term frequency-inverse document frequency and embedding?
Term frequency-inverse document frequency (TF-IDF) is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection. An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
What are the parts of term frequency-inverse document frequency?
Term frequency-inverse document frequency includes inverse document frequency and term frequency.
What is term frequency-inverse document frequency also called?
Term frequency-inverse document frequency is also called TF-IDF.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.











