TF-IDF weights words by combining their frequency within one document with their rarity across a document collection. It provides a simple way to transform text into numerical vectors for similarity matching. section 8.3.1
Calculate the weight in two stages:
- Measure term frequency (TF) by dividing a word’s occurrences in a document by the document’s total word count. In “The sky is blue and beautiful,” blue appears once among six words, giving TF = 0.16. section 8.3.1
- Measure inverse document frequency (IDF) by taking the logarithm of the total document count divided by the count of documents containing the word. section 8.3.1
- Combine TF and IDF so repeated words matter within a document while common collection-wide words receive less weight. section 8.3.1
Vectorizing a document collection
A Python example places eight short documents into a list, creates a TfidfVectorizer, and fits it to produce document vectors. The resulting vectors can then support document similarity comparisons. section 8.3.1
Listing 8.1 document_vector_similarity (transform to vector)
import plotly.graph_objects as go
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = [ #1
"The sky is blue and beautiful.",
"Love this blue and beautiful sky!",
"The quick brown fox jumps over the lazy dog.",
"A king's breakfast has sausages, ham, bacon, eggs, toast, and beans",
"I love green eggs, ham, sausages and bacon!",
"The brown fox is quick and the blue dog is lazy!",
"The sky is very blue and the sky is very beautiful today",
"The dog is lazy but the brown fox is quick!"
]
vectorizer = TfidfVectorizer() #2
X = vectorizer.fit_transform(documents) #3
Inspect the runnable document-vector example, including its sample corpus and scikit-learn vectorizer setup. section 8.3.1
