Term frequency–inverse document frequency (TF-IDF) represents each document as real-valued weights rather than simple word counts. It highlights terms that occur often in one document but appear in relatively few documents, making them useful for relevance ranking, search, spam filtering, and document classification. section 3.0
Build a TF-IDF weight in three stages:
- Calculate term frequency: measure how often a term occurs in a document, usually relative to the document's length. section 3.6.1
- Calculate inverse document frequency: compare the corpus size with the number of documents containing the term; smoothing prevents zero-frequency problems. section 3.6.1
- Combine the two values to give terms greater weight when they are locally frequent and globally informative. section 3.0 section 3.6.1
- Apply logarithmic scaling to reduce Zipf-distribution extremes, so similarly frequent words do not receive vastly different scores. section 3.6.1
Ranking a document corpus
With scikit-learn's TfidfVectorizer, fit a vocabulary on 41,531 Hacker Public Radio sentences and transform them into a sparse 41,531 × 56,404 matrix. Transform the query "where is the lost audio," multiply its sparse vector by the document matrix transpose, and select the highest dot product to retrieve the most relevant sentence. Keeping the matrix sparse avoids allocating roughly 2.3 billion dense values. section 3.6.2 section 3.6.2
>>> log_tf = log(term_occurences_in_doc) -\ ... log(num_terms_in_doc) #1 >>> log_log_idf = log(log(total_num_docs) -\ ... log(num_docs_containing_term)) #2 >>> log_tf_idf = log_tf + log_log_idf #3
Inspect a compact logarithmic TF-IDF implementation that computes log-scaled term frequency, log-scaled inverse document frequency, and their sum. section 3.6.1
Compare normalized TF-IDF vectors with cosine similarity rather than relying on Euclidean distance. Exact spelling remains a boundary: synonyms, paraphrases, and sometimes antonyms can receive misleadingly distant or close vectors. Topic vectors and semantic embeddings address these meaning gaps, while TF-IDF remains useful for full-text search features. section 3.10 section 4.1.1 section 3.8
