liveBook · Manning

Cosine similarity

As explained in LLM Customization and Fine-Tuning

A measure of the angle-based similarity between two vectors, often used to compare weighted text representations.

Cosine similarity compares vectors by their direction rather than their magnitude. It is the dot product of two normalized vectors: a score of 1 means the vectors point in the same direction, 0 means they are orthogonal, and −1 means they point in opposite directions. This makes it useful when vector magnitude could otherwise dominate a comparison. section 4.5.3

The representation determines what a high score means. With sentence embeddings, cosine similarity can rank text chunks by semantic proximity to a query. With TF-IDF vectors, it compares weighted vocabulary: scores near 1 indicate many shared distinctive terms, not necessarily semantic equivalence. section 4.5.3 section 7.1.3

Common uses involve different decisions:

  • Retrieve context: embed the query and text chunks in the same vector space, compare them, and sort for the top-k matches. If vectors are already unit length, their dot product gives the cosine score. section 4.5.3
  • Detect vocabulary drift: build a shared TF-IDF vocabulary, average the reference and production vectors separately, and calculate 1 minus the cosine similarity of the two centroids. A score of 0 means the centroids have the same direction; it does not prove the underlying distributions are identical. section 9.2.2
  • Screen for evaluation contamination: compare training examples with held-out examples using TF-IDF cosine similarity and remove training examples whose maximum score exceeds 0.85. Treat this as a fast overlap screen, not proof of semantic equivalence. section 7.1.3
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
 
# Drop any training example too similar to a held-out test example.
vec = TfidfVectorizer().fit(train_texts + test_texts)
sims = cosine_similarity(vec.transform(train_texts), vec.transform(test_texts))
clean = [ex for ex, s in zip(train_examples, sims) if s.max() <= 0.85]

This scikit-learn example fits a shared TF-IDF vocabulary, compares training and test vectors, and keeps examples at or below the 0.85 overlap threshold. section 7.1.3

For drift detection, sentence-transformer embeddings can replace TF-IDF while retaining the centroid-and-cosine calculation. The richer vectors change what the comparison captures; they do not change the surrounding procedure. section 9.2.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines normalized dot products and works through centroid drift and pairwise overlap screening without a long mathematical derivation.
Examples
Compares query and chunk embeddings, reference and production centroids, and teacher-generated training examples against held-out tests.
What you can do after
Apply it via scikit-learn to screen training examples for excessive TF-IDF overlap with a held-out test set.
How it connects
Connects cosine similarity to sentence embeddings, TF-IDF, vector retrieval, centroids, and data contamination.

Also explained in

Compare how 25 books explain cosine similarity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.