liveBook · Manning

Embedding

As explained in Machines that Think

A learned vector representation of an item whose values encode relationships inferred from data.

An embedding is a learned numerical vector that represents an item through relationships inferred from data. For language, tokens used in similar contexts can acquire similar vectors, placing them near one another in a mathematical space. section 2.5.1

Unlike a one-hot vector, which gives each token a separate sparse representation, a dense embedding uses a compact list of values that can vary gradually across dimensions. The distributional hypothesis provides the intuition: words that appear in similar company often share aspects of meaning. section 2.5.1

Figure 2.6 A simplified illustration of how embeddings represent words as numerical vectors across multiple dimensions. Each row corresponds to a word, and each column represents a property that contributes to its representation. The values indicate how strongly the word is associated with each property, forming a vector that captures patterns of similarity. Words with similar patterns across dimensions, such as apple and pizza or car and robot, end up with similar representations. In real models, these dimensions are learned automatically from data and do not correspond directly to interpretable properties like “edible” or “living.”

A simplified vector illustration shows how patterns across dimensions can represent similarity. Real embedding dimensions are learned and do not necessarily correspond to interpretable properties. section 2.5.1

Embeddings can be learned through prediction. In word2vec, Continuous Bag of Words predicts a word from its surroundings, while Skip-gram predicts surrounding words from a target word. Training adjusts the representations to improve these predictions. The resulting static embeddings give each word one fixed vector, regardless of context. section 2.5.2

Relationships as geometry

Words such as “dog” and “wolf” can occupy nearby regions because they appear in similar contexts. Some relationships also appear as directions: the shift from “walk” to “walked” can resemble the shift from “play” to “played.” section 2.5.4

The resulting semantics are relational: vectors reflect patterns in language use rather than direct contact with the world. Their structure is also difficult to interpret because meaning is distributed across many dimensions. Larger embedding models can require substantial computation for training and inference. section 1.5.6 section 2.5.6

Embeddings also support retrieval-augmented generation. A question and documents are represented as vectors and compared for similarity; selected documents are then inserted as text into the prompt. This helps a system use relevant external information without placing an entire repository in the context window. section 6.4.4

Figure 6.8 Retrieval-augmented generation pipeline. A user query is transformed into an embedding, a vector representation used for similarity comparison against embedded documents in a knowledge base. The system ranks documents by semantic similarity and inserts only the top matches as text into the prompt before generation. The model’s response is thus conditioned on both its internal parameters and selectively retrieved external knowledge.

The retrieval pipeline distinguishes similarity matching with embeddings from inserting the selected document text into the prompt. section 6.4.4

How Machines that Think covers it

How deep it goes
Explains dense vectors, predictive learning and geometric relationships, while noting the limitation of fixed word representations.
Examples
Uses nearby animal words, word-form transformations and question-to-document matching to illustrate embedding relationships.
What you can do after
Evaluate and understand how learned similarity supports retrieval, while recognizing limits of interpretability and grounding.
How it connects
Connects embeddings to the distributional hypothesis, one-hot encoding, transfer learning and retrieval-augmented generation.

Also explained in

Compare how 72 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.