An embedding turns a piece of text into a vector: a list of numbers a model can compute with. Think of it as placing text on a map of meaning rather than filing it only by its exact words. Texts with related meaning tend to occupy nearby positions, while unrelated texts tend to be farther apart. section 1.5 section 4.1
Words as coordinates
In a small illustrative vocabulary, each word can have three coordinates and appear as a point in 3D space. City names such as Rome, Paris, and London cluster together, while functional words such as “the,” “of,” and “is” lie elsewhere. The coordinates—not the word labels—are the units used in later calculations. section 1.5
Figure 4.4 Six pet-related chunks laid in a 2D embedding space
This two-dimensional sketch makes the central intuition visible: chunks near one another represent similar content, even though practical embeddings have many more dimensions. section 4.1
Embeddings serve two closely related jobs. In text generation, tokens are embedded as numbers so an LLM can build context and predict the next token. In retrieval-augmented generation (RAG), document chunks and a user question are embedded into the same kind of space so the system can find relevant chunks by proximity. section 1.5 section 4.5
A practical RAG ingestion and retrieval flow works as follows:
- Read source files and extract their text. section 4.1
- Split the text into manageable chunks, often by paragraphs or with partial overlap. section 4.1
- Convert every chunk into an embedding and save it with metadata in a vector database. section 4.1
- Embed the incoming question, locate the closest chunk vectors, and provide those chunks to the LLM as context. section 4.5
Meaning-based retrieval is the practical advantage. A keyword search primarily matches exact or similar terms; vector retrieval compares embedded semantic content such as intent, topic, and tone. This lets a query retrieve relevant wording even when it does not repeat the document’s exact vocabulary. section 4.1
Embedding size depends on the model. A simple teaching example uses three numbers per token and can be drawn as a chart, whereas the GPT-3 example uses 12,288 numbers per token across a 50,000-token dictionary. High-dimensional vectors are not readily visualized, but the nearby-means-similar mental model remains useful. section 1.6
Embeddings do not by themselves make an LLM know new information. In RAG, they support selecting useful chunks from external documents; the LLM then uses those retrieved chunks when generating an answer. Retrieval quality can be adjusted through choices such as chunk size, overlap, top-k results, and query rewriting. section 4.5
