A token identifier is an index, not a measure of meaning. An embedding gives the token an initial numerical representation: the identifier selects a row from a learned embedding matrix, and that row is a vector. section 12.2.1
Embedding values begin largely at random and are adjusted during training. Backpropagation calculates how the values used in a forward pass contributed to prediction error; an optimization algorithm then updates them along with other model parameters. Across many examples, embeddings capture statistical relationships among token contexts, rather than storing dictionary definitions. section 12.2.1
Embeddings in a pronoun example
A simplified attention calculation represents “trophy,” “box,” and “it” with two-dimensional embeddings [1, 0], [0, 1], and [0.2, 0.8]. Using these vectors directly as queries, keys, and values, the calculation assigns attention weights of approximately 0.39 to “trophy” and 0.61 to “box.” Their weighted combination gives “it” a contextual representation of [0.39, 0.61]. section 12.2.1
This example makes the arithmetic visible, but its vectors do not reproduce the full behavior of a trained transformer. Contextual representations arise within a larger system of attention heads, feed-forward networks, and stacked layers; a model’s capabilities cannot be attributed to one attention calculation. section 12.2







































































