An embedding is a numerical vector that represents the meaning of a piece of text. It lets software compare documents using vector distance: nearby vectors are intended to represent similar content. This makes ordinary vector-based clustering useful for tasks such as grouping related emails. section 4.4.1
Figure 4.3 Clustering emails. We first calculate embedding vectors for all emails. Then we cluster those vectors to assign emails with similar content to the same cluster.
The diagram shows the two-step workflow: calculate a vector for each email, then cluster the vectors. section 4.4.1
To create one, send text to an embedding model rather than a chat-completions model. The response contains the vector, model name, and token usage. Vector size depends on the model; the example model produces a vector with more than 1,000 dimensions. section 4.4.2
Get a text embedding in Python
Use `client = openai.OpenAI()`, then `response = client.embeddings.create(model='text-embedding-ada-002', input=text)`. Read the vector from `response.data[0].embedding`. For a large collection, retry occasional failed requests before giving up. section 4.4.2
Once texts have vectors, compare them for different jobs:
- Clustering: group documents with similar content. One trial clusters a mixed collection of emails and poems into two groups. section 4.4.1 section 4.4.5
- Retrieval: embed documents once, embed each new question, and find documents with nearby vectors to help answer it. section 4.4.6
- Outlier detection: compare a document's vector with the others to find unusually different texts. section 4.4.6
LlamaIndex can handle much of a retrieval pipeline: load files, split long documents into chunks, index them with embeddings, and retrieve relevant chunks for a question. Similar vectors suggest related topics; they do not, by themselves, establish that a retrieved document answers the question. section 10.6.1 section 10.6.3
