An embedding is a learned vector: an ordered list of numbers used to represent an item inside a model. A vector is one-dimensional; put one vector per row and the result is a matrix. The numbers are learned coordinates, not labels a person assigns to features by hand. section 5.1 section 6.3
Why learn the coordinates? Hand-designed image features such as HOG could make an irrelevant pattern of rippled water look car-like. A classifier might then separate its inputs faithfully while still making the wrong prediction: the representation had already confused the signal. Learned features offer a way to revise what the model measures, though they depend on data and are harder to interpret. section 2.1
For text, an embedding layer is a learned lookup table. Tokenization first turns text into vocabulary IDs, which carry no meaning by themselves. Looking up each ID replaces it with a vector. The original Transformer used 512 numbers per token; a sequence of tokens therefore became a matrix with one row per token and 512 columns. Attention could then combine those representations to make each token’s representation sensitive to its context. section 5.1
A word-vector analogy
Word2vec, introduced by Tomas Mikolov, Ilya Sutskever, and colleagues in 2013, learned word vectors from large text collections. Words used in similar contexts could lie close together. The illustrative operation Paris − France + Italy can point toward Rome, or toward a vector closer to Rome than to unrelated words. It shows a useful regularity in learned coordinates, not proof that vector arithmetic amounts to human-like reasoning. section 5.3
Figure 5.4 Word-embedding vector operations: Paris – France + Italy = Rome
The diagram makes the Paris − France + Italy = Rome analogy visible as operations on word-embedding vectors. section 5.3
Not every vector in a language model plays the same role. An RNN’s changing hidden state carries context forward through a sequence. An early encoder–decoder instead compressed an entire input sentence into one fixed-length context or “thought vector” for its decoder. That created a bottleneck as sentences grew: a useful vector can still lose details when asked to hold too much. Attention offered a way to draw selectively on representations rather than depend on that single compressed summary. section 4.1 section 5.3
