An embedding is a vector—an ordered list of numbers—that represents information a model can use. For example, a token embedding represents a piece of text by its position in a space of meaning. Not every vector is an embedding: vectors can also hold weights, positions, or other numerical values. section 11.12 section 9.13 section 9.20
Imagine a grocery store as a map of meaning: apples and bananas are close together, while apples and laundry detergent are far apart. An embedding is like an item’s coordinates on that map. Token IDs alone do not provide this relationship: neighboring IDs need not have related meanings. section 11.12
A Transformer can look up each token ID in an embedding matrix, a trainable table with one row per vocabulary item. The embedding width is the number of values in each row. Values begin randomly and are adjusted during training, allowing the representation to improve. section 11.14 section 11.11
Adding token and position information
With width 4, suppose a token’s embedding is [0.003, -0.007, 0.012, 0.001] and its position vector is [0, 1, 0, 1]. Add corresponding values to get [0.003, 0.993, 0.012, 1.001]. The resulting input vector carries both token and position information into the Transformer. section 11.16
Table 11.10 A subset of tokens and their embedding added to the positional encodings
|
Position |
Token ID |
Token |
x0 |
x1 |
x2 |
x3 |
| E for 0 |
0 |
a |
0.003 |
-0.007 |
0.012 |
0.001 |
| P for 0 |
|
|
0 |
1 |
0 |
1 |
| X for 0 |
|
|
0.0030 |
0.9930 |
0.0120 |
1.0010 |
| E for 1 |
|
l |
-0.008 |
0.005 |
-0.002 |
0.009 |
| P for 1 |
|
|
0.8415 |
0.5403 |
0.0100 |
1.0000 |
| X for 1 |
|
|
0.8335 |
0.5453 |
0.0080 |
1.0090 |
| … |
||||||
| E for 31 |
6 |
b |
0.7381 |
-0.2512 |
-0.6719 |
0.7938 |
| P for 31 |
|
|
-0.4040 |
0.9147 |
0.3051 |
0.9523 |
| X for 31 |
|
|
0.3341 |
0.6635 |
-0.3668 |
1.7461 |
Compare embedding, position, and combined vectors for several tokens. section 11.16
Embeddings can also guide image generation. A text encoder turns a label such as “a happy face” into a vector, which a feed-forward network transforms into guidance for the image model. In one simple U-Net design, the text and timestep vectors are added and injected into the bridge; larger designs can use cross-attention to connect image features with text. section 12.11 section 12.18 section 12.29

















































