To find useful passages for a user’s question, an app needs a way to compare the question with its knowledge base. A text embedding model converts each piece of text into a numerical vector that captures aspects of its meaning. The app can then look for nearby vectors instead of relying only on matching words. section 10.5.1
Picture a movie catalog
Imagine placing movies on a chart according to comedic value and explosions per hour. Similar movies would appear near one another. Text embeddings use many more dimensions—hundreds or thousands—and those dimensions are not usually human-readable labels. section 10.5.1
For retrieval-augmented generation, embed the user’s message and the knowledge-base chunks, then find the most similar chunks. A vector database such as Pinecone stores embeddings and searches for close matches; cosine similarity scores similarity by the angle between vectors. Here, “vectorization” means representing text as vectors. It is distinct from a pandas vectorized calculation, such as checking every value in a column with `.isin()`. section 10.5.1 section 6.3.1
