Cosine similarity compares vectors by their direction rather than their magnitude. It is the dot product of two normalized vectors: a score of 1 means the vectors point in the same direction, 0 means they are orthogonal, and −1 means they point in opposite directions. This makes it useful when vector magnitude could otherwise dominate a comparison. section 4.5.3
The representation determines what a high score means. With sentence embeddings, cosine similarity can rank text chunks by semantic proximity to a query. With TF-IDF vectors, it compares weighted vocabulary: scores near 1 indicate many shared distinctive terms, not necessarily semantic equivalence. section 4.5.3 section 7.1.3
Common uses involve different decisions:
- Retrieve context: embed the query and text chunks in the same vector space, compare them, and sort for the top-k matches. If vectors are already unit length, their dot product gives the cosine score. section 4.5.3
- Detect vocabulary drift: build a shared TF-IDF vocabulary, average the reference and production vectors separately, and calculate 1 minus the cosine similarity of the two centroids. A score of 0 means the centroids have the same direction; it does not prove the underlying distributions are identical. section 9.2.2
- Screen for evaluation contamination: compare training examples with held-out examples using TF-IDF cosine similarity and remove training examples whose maximum score exceeds 0.85. Treat this as a fast overlap screen, not proof of semantic equivalence. section 7.1.3
from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity # Drop any training example too similar to a held-out test example. vec = TfidfVectorizer().fit(train_texts + test_texts) sims = cosine_similarity(vec.transform(train_texts), vec.transform(test_texts)) clean = [ex for ex, s in zip(train_examples, sims) if s.max() <= 0.85]
This scikit-learn example fits a shared TF-IDF vocabulary, compares training and test vectors, and keeps examples at or below the 0.85 overlap threshold. section 7.1.3
For drift detection, sentence-transformer embeddings can replace TF-IDF while retaining the centroid-and-cosine calculation. The richer vectors change what the comparison captures; they do not change the surrounding procedure. section 9.2.2
























