TF-IDF gives words weights based on how often they appear in a document and how distinctive they are across a collection. It turns each document into a sparse vector: positions represent words, and weights reflect their importance. It measures word use, not meaning. section 6.1.3 section 6.2.1
Consider the word “blue” in “The sky is blue and beautiful.” Term frequency captures its occurrence in that sentence; inverse document frequency offsets that signal according to how common “blue” is across the collection. A word repeated in a document is not automatically distinctive if it also appears throughout the collection. section 6.1.3
Listing 6.1 document_vector_similarity (transform to vector)
import plotly.graph_objects as go
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = [ #1
"The sky is blue and beautiful.",
"Love this blue and beautiful sky!",
"The quick brown fox jumps over the lazy dog.",
"A king's breakfast has sausages, ham, bacon, eggs, toast, and beans",
"I love green eggs, ham, sausages and bacon!",
"The brown fox is quick and the blue dog is lazy!",
"The sky is very blue and the sky is very beautiful today",
"The dog is lazy but the brown fox is quick!"
]
vectorizer = TfidfVectorizer() #2
X = vectorizer.fit_transform(documents) #3
The Python example passes a small collection of sentences to scikit-learn’s TfidfVectorizer. Its fit_transform call produces the document vectors used for comparison. section 6.1.3
Cosine similarity can then compare those vectors. Try sentences about a blue sky, a fox and dog, and breakfast foods: shared distinctive words help bring documents together. Inspecting the scores is more useful than assuming that a high-scoring match understands the sentence. section 6.1.3 section 6.5
Choose TF-IDF when exact wording matters, such as a name, code, or rare keyword. A search for “vehicles” will not match a document that says only “cars” merely because the words are related. Semantic embeddings address meaning-based matching; hybrid retrieval can combine that signal with exact-keyword search. section 6.2.1
