Vectorization transforms text into numerical vectors so documents can be compared computationally. The chosen method shapes how semantic similarity is measured. section 8.3.1
A practical document-similarity workflow follows these steps:
- Collect a set of documents, such as sentences about skies, animals, or food. section 8.3.1
- Create a TfidfVectorizer and call fit_transform to convert the documents into vectors. section 8.3.1
- Calculate cosine similarities, then inspect how closely a selected document matches the others. section 8.3.1
Comparing vectorization methods
A semantic-search exercise replaces TF–IDF with Word2Vec or BERT embeddings, runs searches, and compares the results to understand differences in effectiveness. section 8.8
Listing 8.1 document_vector_similarity (transform to vector)
import plotly.graph_objects as go
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = [ #1
"The sky is blue and beautiful.",
"Love this blue and beautiful sky!",
"The quick brown fox jumps over the lazy dog.",
"A king's breakfast has sausages, ham, bacon, eggs, toast, and beans",
"I love green eggs, ham, sausages and bacon!",
"The brown fox is quick and the blue dog is lazy!",
"The sky is very blue and the sky is very beautiful today",
"The dog is lazy but the brown fox is quick!"
]
vectorizer = TfidfVectorizer() #2
X = vectorizer.fit_transform(documents) #3
The runnable listing shows the document data, TF–IDF vectorizer, and transformation step used before similarity calculations. section 8.3.1
