Suppose two documents use the same mix of words, but one is much longer. Cosine similarity treats their vector representations as similar because it compares direction rather than magnitude. It measures the cosine of the angle between two nonzero vectors. Unlike Euclidean distance or a dot product, it does not let document length dominate the comparison. section 6.1.3
Three useful distinctions for document search:
- Similarity measures likeness: a value of 1 indicates identical direction, while –1 indicates opposite direction. section 6.1.3
- Cosine distance reverses the scale: distance = 1 – similarity. A distance of 0 marks the most similar result; smaller distances are better matches. section 6.2.2
- With sparse TF-IDF vectors, a few shared terms can produce a misleading match. Dense semantic embeddings offer another way to represent documents for retrieval. section 6.1.3
Compare sentences
Turn a small collection of sentences about a blue sky, a fox, and breakfast into TF-IDF vectors. Compute their cosine-similarity matrix, then select one sentence to inspect its scores against the rest. Try the queries “blue” and “breakfast” to see how the matches change. Replacing TF-IDF with semantic embeddings makes the rankings comparable across two representations. section 6.1.3 section 6.5
Listing 6.2 document_vector_similarity (cosine similarity)
cosine_similarities = cosine_similarity(X) #1
while True: #2
selected_document_index = input(f"Enter a document number
➥ (0-{len(documents)-1}) or 'exit' to quit: ").strip()
if selected_document_index.lower() == 'exit':
break
if not selected_document_index.isdigit() or
➥ not 0 <= int(selected_document_index) < len(documents):
print("Invalid input. Please enter a valid document number.")
continue
selected_document_index = int(selected_document_index) #3
selected_document_similarities = cosine_similarities[selected_document_index] #4
# code to plot document similarities omitted
The cosine_similarity(X) call builds the comparison matrix; selecting a document index retrieves that document’s scores. Input checks keep the index within the document collection. section 6.1.3
Listing 6.5 document_query_chromadb.py (relevant code sections)
embeddings = [get_embedding(doc) for doc in documents] #1
ids = [f"id{i}" for i in range(len(documents))] #2
chroma_client = chromadb.Client() #2
collection = chroma_client.create_collection( #2
name="documents") #2
collection.add( #3
embeddings=embeddings,
documents=documents,
ids=ids
)
def query_chromadb(query, top_n=2): #4
query_embedding = get_embedding(query)
results = collection.query(
query_embeddings=[query_embedding],
n_results=top_n
)
return [(id, score, text) for id, score, text in
zip(results['ids'][0],
results['distances'][0],
results['documents'][0])]
while True: #5
query = input("Enter a search query (or 'exit' to stop): ")
if query.lower() == 'exit':
break
top_n = int(input("How many top matches do you want to see? "))
search_results = query_chromadb(query, top_n)
print("Top Matched Documents:")
for id, score, text in search_results:
print(f"""
ID:{id} TEXT: {text} SCORE: {round(score, 2)}
""") #5
print("\n")
###Output
Enter a search query (or 'exit' to stop): dogs are lazy
How many top matches do you want to see? 3
Top Matched Documents:
ID:id7 TEXT: The dog is lazy but the brown fox is quick! SCORE: 0.24
ID:id5 TEXT: The brown fox is quick and the blue dog is lazy! SCORE: 0.28
ID:id2 TEXT: The quick brown fox jumps over the lazy dog. SCORE: 0.29
For embedding search, the Chroma DB query returns document IDs, text, and distances rather than similarity scores. Read a lower cosine distance as a closer match. section 6.2.2
