Cosine similarity compares the direction of two vectors rather than their length. For text embeddings, this helps find documents that resemble a query even when the documents differ in length. Its value ranges from −1 for opposite directions to 1 for the same direction; it is less useful when vector magnitude matters. section 7.5.2
In semantic search, a query and candidate passages are converted to embeddings, then compared to find relevant results. Cosine similarity can serve as the comparison metric in a vector index or retriever. It is one option, not a universal default: choose a metric that fits the data, query, and underlying model. section 7.4 section 7.5.2
Related measures make different trade-offs:
- Squared Euclidean distance measures separation between points, making it useful when geometric distance matters. section 7.5.2
- Dot product also reflects vector magnitude. For normalized embeddings, it is equivalent to cosine similarity and can be more efficient. section 7.5.2
Finding similar blog posts
A Redis index can declare an embedding field with HNSW, FLOAT32 vectors, 1536 dimensions, and DISTANCE_METRIC set to COSINE. A search application embeds a question, runs a K-nearest-neighbors query against that field, and returns the top three matching posts. The query embedding must match the index’s configured dimensions. section 8.3 section 8.5
Table 7.1 Choosing the right distance measure
|
Measure
|
When to use
|
Advantage
|
Disadvantage
|
|---|---|---|---|
|
Cosine similarity
|
Ideal for text and document similarity, where the magnitude of the vectors is not as important as the orientation; common in NLP tasks
|
Effective in high-dimensional spaces and for normalized vectors; ignores the magnitude of vectors, focusing on orientation, making it suitable for comparing documents of different lengths
|
It is not effective if the magnitude of vectors is important.
|
|
Squared Euclidean (L2)
|
Suitable for geometric or spatial data, like in image processing or when clustering multi-dimensional numerical data
|
It reflects the actual distance between points in a Euclidean space, making it intuitive and suitable for spatial datasets.
|
It can be sensitive to the scale of the data. High dimensions can lead to the curse of dimensionality.
|
|
Dot product
|
Efficient for high-volume, high-dimensional data, such as user preferences in recommendation systems
|
Computationally efficient, especially for sparse vectors. It is good for cases where the magnitude of vectors matters.
|
Interpretation is less intuitive than cosine similarity and can be sensitive to vector magnitudes.
|
|
Hamming distance
|
Best for comparing binary or categorical data, such as genetic sequences or error detection in data transmission
|
Simple and effective for datasets with discrete attributes
|
It only applies to strings of equal length and doesn’t consider the magnitude of differences.
|
|
Manhattan (L1) distance
|
Useful in grid-like pathfinding (e.g., urban road layouts) and in cases where differences in individual dimensions are important
|
It is more sensitive to differences in individual dimensions than L2 distance; robust to outliers.
|
It may not reflect the true distance in nongrid-like spaces or high-dimensional data.
|
Compare when cosine similarity, squared Euclidean distance, and dot product suit a search task, including cosine similarity’s limitation when magnitude matters. section 7.5.2
