liveBook · Manning

Embedding

As explained in Grokking AI Algorithms, Second Edition

An input vector is an ordered one-dimensional array of values supplied to a computational layer.

An embedding is a vector—an ordered list of numbers—that represents information a model can use. For example, a token embedding represents a piece of text by its position in a space of meaning. Not every vector is an embedding: vectors can also hold weights, positions, or other numerical values. section 11.12 section 9.13 section 9.20

Imagine a grocery store as a map of meaning: apples and bananas are close together, while apples and laundry detergent are far apart. An embedding is like an item’s coordinates on that map. Token IDs alone do not provide this relationship: neighboring IDs need not have related meanings. section 11.12

A Transformer can look up each token ID in an embedding matrix, a trainable table with one row per vocabulary item. The embedding width is the number of values in each row. Values begin randomly and are adjusted during training, allowing the representation to improve. section 11.14 section 11.11

Adding token and position information

With width 4, suppose a token’s embedding is [0.003, -0.007, 0.012, 0.001] and its position vector is [0, 1, 0, 1]. Add corresponding values to get [0.003, 0.993, 0.012, 1.001]. The resulting input vector carries both token and position information into the Transformer. section 11.16

Table 11.10 A subset of tokens and their embedding added to the positional encodings

Position

Token ID

Token

x0

x1

x2

x3

E for 0

0

a

0.003

-0.007

0.012

0.001

P for 0

 

 

0

1

0

1

X for 0

 

 

0.0030

0.9930

0.0120

1.0010

E for 1

 

l

-0.008

0.005

-0.002

0.009

P for 1

 

 

0.8415

0.5403

0.0100

1.0000

X for 1

 

 

0.8335

0.5453

0.0080

1.0090

…

E for 31

6

b

0.7381

-0.2512

-0.6719

0.7938

P for 31

 

 

-0.4040

0.9147

0.3051

0.9523

X for 31

 

 

0.3341

0.6635

-0.3668

1.7461

Compare embedding, position, and combined vectors for several tokens. section 11.16

Embeddings can also guide image generation. A text encoder turns a label such as “a happy face” into a vector, which a feed-forward network transforms into guidance for the image model. In one simple U-Net design, the text and timestep vectors are added and injected into the bridge; larger designs can use cross-attention to connect image features with text. section 12.11 section 12.18 section 12.29

How Grokking AI Algorithms, Second Edition covers it

How deep it goes
Explains trainable lookup matrices, elementwise positional addition, text projection, and embedding injection with numerical examples and learning-oriented code.
Examples
Traces a four-value token embedding combined with position and a text-label embedding used to guide a happy-face image.
What you can do after
Evaluate and understand how embeddings are formed, combined with position, and used to guide model computations.
How it connects
Connects embeddings to tokenization, positional encoding, self-attention, feed-forward networks, U-Nets, and cross-attention.

Also explained in

Compare how 50 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.