Term frequency-inverse document frequency

As explained in Math and Architectures of Deep Learning

A document-representation weighting scheme that combines term frequency with inverse document frequency to emphasize discriminative words.

Term frequency–inverse document frequency (TF-IDF) represents documents as weighted vectors for retrieval. It combines how often a term appears in one document with how uncommon that term is across the document collection. section 4.6.1 section 4.7

Build the representation in three steps:

• Count a term’s occurrences in each document to obtain its term frequency (TF). Case variants, singular/plural forms, and related word stems are typically mapped to one term. section 4.6.1
• Down-weight widespread terms with inverse document frequency (IDF): invert the absolute logarithm of the fraction of documents containing the term. Common terms receive low weights; relatively esoteric terms receive high weights. section 4.6.1
• Multiply TF by IDF for every vocabulary term. Each document becomes a point in a vocabulary-sized space, usually stored as a sparse vector rather than explicitly materialized. section 4.6.1

Why weighting matters

The word “the” appears in many documents, so its IDF is very low and contributes little to matching. If “gun” occurs zero times in d0 and d6, once in d1, and three times in d3, its TF changes across those documents; multiplying by IDF preserves more discriminative occurrences. section 4.6.1

TF-IDF supports document matching and retrieval through cosine similarity. It compares documents as vectors, while latent semantic analysis can apply PCA to those vectors to uncover topic-level similarities beyond directly overlapping terms. section 4.7

Equation 4.13

Use the displayed equation to inspect the formal TF-IDF weighting relationship. section 4.6.1

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines TF, IDF, vector construction, and sparse storage, then connects the representation to cosine similarity and PCA-based topic modeling.

Examples

Uses “the” as a common-term example and contrasts gun counts across documents d0, d1, d3, and d6.

What it assumes

Defines terms, word counts, vocabulary, and document vectors before introducing the weighting scheme.

What you can do after

Apply TF-IDF with cosine similarity to match and retrieve documents.

How it connects

Connects term frequency and inverse document frequency to cosine similarity, sparse vectors, document retrieval, PCA, and latent semantic analysis.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA TF-IDF model represents text using term-frequency and inverse-document-frequency weights for terms in documents.
• 

  liveBookmanning.com
  AI Agents in Action, Second EditionTF-IDF is a term-weighting technique that scores a word by its frequency in a document relative to its frequency across a collection of documents.
• 

  liveBookmanning.com
  Elasticsearch in Action, Second EditionA term frequency–inverse document frequency algorithm ranks documents using query-term frequency in documents and term rarity across the collection.
• 

  liveBookmanning.com
  AI-Powered SearchTF-IDF is a text-weighting metric that combines term frequency and inverse document frequency to estimate a term's importance within a document collection.
• 

  liveBookmanning.com
  Data Without LabelsTerm frequency-inverse document frequency (TF-IDF) assigns a term a weight based on its frequency in a document and its inverse frequency across a document collection.
• 

  liveBookmanning.com
  AI Agents in ActionTF-IDF is a term-weighting method that scores words by their frequency in a document relative to their frequency across a document collection.
• 

  liveBookmanning.com
  Machine Learning System DesignA text-weighting representation that scores terms by their frequency in a document relative to their frequency across documents.
• 

  liveBookmanning.com
  Generative AI in ActionTF-IDF is a statistical weighting scheme that scores a term according to its frequency in a document and its rarity across a document collection.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthTF-IDF is a term-weighting technique that combines a term's frequency in a document with the inverse frequency of documents containing that term.

Compare how 10 books explain term frequency-inverse document frequency →

Related concepts

• inverse document frequencyhas part · Inverse document frequency is a corpus-based measure of term rarity that assigns lower weights to terms appearing in many documents and higher weights to rarer terms.
• term frequencyhas part · Term frequency is the raw or normalized count of a term’s occurrences within a document or other text unit.
• lexical searchused by · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
• embeddingcontrasts with · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• BM25alternative to · BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
• Word2Vecalternative to · Word2Vec is a family of shallow neural models that learns vector representations of words from their contexts in text.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.