Vectorization turns documents or queries into numerical vectors so they can be compared. An exact database lookup finds a matching value; vector similarity search ranks items by how similar their vectors are. In retrieval-augmented generation (RAG), this helps find document chunks relevant to a question before a language model writes an answer. section 6.1.1
A practical progression has three steps:
- Vectorize a small document set with TF-IDF, then use cosine similarity to compare each document with the others. A document scores 1 when compared with itself. section 6.1.3
- Store the document vectors and vectorize each new query with the same fitted vectorizer. Rank the results by similarity score and return the highest matches. section 6.2
- For RAG, split a larger document into chunks, embed the chunks, and store them in a vector database. Embed a query to retrieve relevant chunks; a separate language model uses them to generate the answer. section 6.1.1 section 6.3.2
From comparison to script search
Start with short sentences about a blue sky, a fox, and a dog. TF-IDF turns them into vectors, and cosine similarity shows which documents score closest. A larger example splits a movie script into chunks, embeds them with an OpenAI embedding model, and stores them in Chroma DB for a search tool. The choice of vectorization matters: the simple TF-IDF search can match words and phrases while missing context and meaning. section 6.1.3 section 6.3.2 section 6.2
Listing 6.3 document_vector_database.py
# code above omitted
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)
vector_database = X.toarray() #1
def cosine_similarity_search(query,
database,
vectorizer,
top_n=5): #2
query_vec = vectorizer.transform([query]).toarray()
similarities = cosine_similarity(query_vec, database)[0]
top_indices = np.argsort(-similarities)[:top_n] # Top n indices
return [(idx, similarities[idx]) for idx in top_indices]
while True: #3
query = input("Enter a search query (or 'exit' to stop): ")
if query.lower() == 'exit':
break
top_n = int(input("How many top matches do you want to see? "))
search_results = cosine_similarity_search(query,
vector_database,
vectorizer,
top_n)
print("Top Matched Documents:")
for idx, score in search_results:
print(f"- {documents[idx]} (Score: {score:.4f})") #4
print("\n")
###Output
Enter a search query (or 'exit' to stop): blue
How many top matches do you want to see? 3
Top Matched Documents:
- The sky is blue and beautiful. (Score: 0.4080)
- Love this blue and beautiful sky! (Score: 0.3439)
- The brown fox is quick and the blue dog is lazy! (Score: 0.2560)
The in-memory search listing shows the key boundary: fit TF-IDF on documents, transform a query with that vectorizer, then rank document vectors by cosine similarity. section 6.2
Retrieval alone does not guarantee a grounded answer. When retrieved chunks supply the context, instruct the language model to answer from that context and say when it does not contain the answer. section 6.3.2
