liveBook · Manning

Embedding

As explained in Timeless Algorithms, The Foundational Ideas

An embedding is a numerical vector representation of an item, such as a word or token, in a learned space.

A token identifier is an index, not a measure of meaning. An embedding gives the token an initial numerical representation: the identifier selects a row from a learned embedding matrix, and that row is a vector. section 12.2.1

Embedding values begin largely at random and are adjusted during training. Backpropagation calculates how the values used in a forward pass contributed to prediction error; an optimization algorithm then updates them along with other model parameters. Across many examples, embeddings capture statistical relationships among token contexts, rather than storing dictionary definitions. section 12.2.1

Embeddings in a pronoun example

A simplified attention calculation represents “trophy,” “box,” and “it” with two-dimensional embeddings [1, 0], [0, 1], and [0.2, 0.8]. Using these vectors directly as queries, keys, and values, the calculation assigns attention weights of approximately 0.39 to “trophy” and 0.61 to “box.” Their weighted combination gives “it” a contextual representation of [0.39, 0.61]. section 12.2.1

This example makes the arithmetic visible, but its vectors do not reproduce the full behavior of a trained transformer. Contextual representations arise within a larger system of attention heads, feed-forward networks, and stacked layers; a model’s capabilities cannot be attributed to one attention calculation. section 12.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines embedding lookup and training, then traces simplified vectors through an attention calculation rather than reproducing a complete trained transformer.
Examples
Uses two-dimensional vectors for “trophy,” “box,” and “it” in a numerical pronoun-resolution example.
What you can do after
Evaluate how token embeddings contribute to a simplified attention calculation without mistaking it for a complete transformer.
How it connects
Connects embeddings to tokenization, backpropagation, self-attention, softmax, and next-token prediction.

Also explained in

Compare how 72 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.