TF-IDF represents documents by weighting terms according to both their local frequency and their rarity across the document collection. It emphasizes words that distinguish documents rather than words that merely occur often. In a medical corpus, “diabetes” may identify relevant documents more effectively than the common word “disease.” section 7.10
Term frequency (TF) counts a term within a document, while inverse document frequency (IDF) uses the logarithm of the ratio between the total document count and the number of documents containing that term. TF-IDF is their product, combining document-specific frequency with collection-wide rarity. section 7.10
Sports-document calculation
For a 100-word sports document containing “ground” five times and “backhand” twice, the term frequencies are 0.05 and 0.02. Because “ground” is common in sports while “backhand” is rarer, inverse document frequency can give the latter greater relative importance. section 7.10
A practical workflow uses scikit-learn's TfidfVectorizer to convert text into a matrix, optionally removing English stop words. The resulting vectors can feed K-means clustering, whose centroids and prominent terms help separate football and travel documents. section 7.16
tfidf_vectorizer = TfidfVectorizer(stop_words='english') X = tfidf_vectorizer.fit_transform(text)
Inspect the Python statement that fits a TF-IDF vectorizer to text and produces the document-term matrix. section 7.16
