A sparse vector stores values across many positions while leaving most positions at zero or otherwise absent. In text retrieval, TF-IDF creates sparse vectors from the words appearing in documents and their distinctiveness across a collection. section 6.1.3
Use this contrast to place sparse vectors in a retrieval pipeline:
- Treat TF-IDF as a term-counting representation: it measures term importance, not the meaning of the text. section 6.1.3
- Expect vocabulary-sensitive matching: a query about “vehicles” does not match a document about “cars” when the words differ. section 6.1.3
- Contrast sparse vectors with dense vectors, whose dimensions carry distributed semantic information and support meaning-based matches. section 6.1.3 section 6.1.1
A small document collection
Given documents about blue skies, foxes, dogs, and breakfasts, TfidfVectorizer converts the collection into a sparse matrix. Cosine similarity can then compare document vectors and rank documents with more similar word patterns. section 6.1.3
Listing 6.1 document_vector_similarity (transform to vector)
import plotly.graph_objects as go
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = [ #1
"The sky is blue and beautiful.",
"Love this blue and beautiful sky!",
"The quick brown fox jumps over the lazy dog.",
"A king's breakfast has sausages, ham, bacon, eggs, toast, and beans",
"I love green eggs, ham, sausages and bacon!",
"The brown fox is quick and the blue dog is lazy!",
"The sky is very blue and the sky is very beautiful today",
"The dog is lazy but the brown fox is quick!"
]
vectorizer = TfidfVectorizer() #2
X = vectorizer.fit_transform(documents) #3
Inspect the smallest working implementation: define documents, create TfidfVectorizer, and transform the collection into vectors. section 6.1.3
