An embedding is a list of numbers that gives a token or text label a form a neural network can use. The list is a vector: a one-dimensional array such as [0.003, -0.007, 0.012, 0.001]. Think of an embedding as a location in a huge store of meaning. Related items can end up near one another, while unrelated items can be far apart. section 9.13 section 11.12
Why not give the network a token ID directly? IDs are labels, not meanings: ID 10 is not inherently closer to ID 11 than to ID 17. An embedding matrix solves this by storing a vector for each token ID. Its number of rows follows the vocabulary size; its width determines how many values describe each token. The vectors start with random values and change during training. section 11.12 section 11.14
From token to Transformer input
Suppose the token “a” has embedding [0.003, -0.007, 0.012, 0.001]. At position 0, its positional encoding is [0, 1, 0, 1]. Add matching entries to get [0.003, 0.993, 0.012, 1.001]. The resulting four-value input carries both the token’s identity and its place in the sequence. Repeating this for a 32-token batch produces a 32×4 matrix for the Transformer. section 11.16
Table 11.10 A subset of tokens and their embedding added to the positional encodings
|
Position |
Token ID |
Token |
x0 |
x1 |
x2 |
x3 |
| E for 0 |
0 |
a |
0.003 |
-0.007 |
0.012 |
0.001 |
| P for 0 |
|
|
0 |
1 |
0 |
1 |
| X for 0 |
|
|
0.0030 |
0.9930 |
0.0120 |
1.0010 |
| E for 1 |
|
l |
-0.008 |
0.005 |
-0.002 |
0.009 |
| P for 1 |
|
|
0.8415 |
0.5403 |
0.0100 |
1.0000 |
| X for 1 |
|
|
0.8335 |
0.5453 |
0.0080 |
1.0090 |
| … |
||||||
| E for 31 |
6 |
b |
0.7381 |
-0.2512 |
-0.6719 |
0.7938 |
| P for 31 |
|
|
-0.4040 |
0.9147 |
0.3051 |
0.9523 |
| X for 31 |
|
|
0.3341 |
0.6635 |
-0.3668 |
1.7461 |
Compare each token’s embedding, positional encoding, and element-by-element sum. section 11.16
Embeddings also help guide image generation. A text encoder can turn a label such as “a happy face” into one vector. A feed-forward network adapts that general text embedding into guidance for a denoising model. In a simple U-Net, a 16-value text embedding is added to a timestep embedding; each value of the combined vector is then added across its corresponding bridge channel. Larger models can instead use cross-attention to connect text with image features. section 12.11 section 12.18 section 12.29
An embedding is not a fixed code for meaning. Its values depend on training and on the model that uses it. The practical pattern is to turn a discrete label into a suitably sized numeric vector, then combine or transform that vector where the network needs it. section 11.14 section 11.16 section 12.11
