Inverse document frequency (IDF) measures how uncommon a term is across a document collection. Within TF-IDF, it gives more weight to distinctive words and less to words appearing throughout the corpus. section 6.1.3
Use this mental model:
- Term frequency asks how often a word appears in one document. section 6.1.3
- IDF offsets common words by considering their frequency across the document set. section 6.1.3
- TF-IDF measures term importance, not meaning: “vehicles” does not match “cars” merely because they are synonymous. section 6.1.3
Listing 6.1 document_vector_similarity (transform to vector)
import plotly.graph_objects as go
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = [ #1
"The sky is blue and beautiful.",
"Love this blue and beautiful sky!",
"The quick brown fox jumps over the lazy dog.",
"A king's breakfast has sausages, ham, bacon, eggs, toast, and beans",
"I love green eggs, ham, sausages and bacon!",
"The brown fox is quick and the blue dog is lazy!",
"The sky is very blue and the sky is very beautiful today",
"The dog is lazy but the brown fox is quick!"
]
vectorizer = TfidfVectorizer() #2
X = vectorizer.fit_transform(documents) #3
Inspect a minimal Python example that transforms a document collection with scikit-learn’s TfidfVectorizer before comparing document vectors. section 6.1.3
