Term frequency (TF) measures how often a term appears in a document. It is commonly normalized by the document’s total word count, making documents of different lengths easier to compare. section 6.1.3
TF counts occurrence rather than meaning. A TF-IDF search can connect documents sharing the word “blue,” but will not match “vehicles” with “cars”; semantic embeddings address that vocabulary gap. section 6.1.3
A small document collection
For “The sky is blue and beautiful,” focusing on “blue” illustrates term frequency directly. A larger example compares sentences about skies, foxes, dogs, eggs, ham, and bacon using TF-IDF vectors. section 6.1.3
Listing 6.1 document_vector_similarity (transform to vector)
import plotly.graph_objects as go
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = [ #1
"The sky is blue and beautiful.",
"Love this blue and beautiful sky!",
"The quick brown fox jumps over the lazy dog.",
"A king's breakfast has sausages, ham, bacon, eggs, toast, and beans",
"I love green eggs, ham, sausages and bacon!",
"The brown fox is quick and the blue dog is lazy!",
"The sky is very blue and the sky is very beautiful today",
"The dog is lazy but the brown fox is quick!"
]
vectorizer = TfidfVectorizer() #2
X = vectorizer.fit_transform(documents) #3
Inspect a minimal Python setup that uses TfidfVectorizer to transform a collection of documents into sparse numerical vectors. section 6.1.3
