A token is a discrete text unit in a model’s vocabulary. It may represent a whole word, part of a word, punctuation, or another recurring text element. Tokenization turns input text into a sequence of integer identifiers; applying the same tokenizer to the same text produces the same identifiers. section 12.2.1
An identifier is an index, not a numerical measure of meaning. It selects a row from a learned embedding matrix, giving the token its initial vector representation. In a transformer, positional information supplements token embeddings so that the model can distinguish sequence order. section 12.2.1 section 12.1.2
Self-attention makes token representations sensitive to context. For each token, the model weights information from other relevant tokens and combines it into a new representation. Tokens far apart can interact directly rather than relying on information carried through every intervening recurrent state. The resulting representations support a probability distribution over possible next tokens. section 12.1.2 section 12.2.1
A pronoun in context
In “The trophy didn’t fit in the box because it was too…”, a continuation depends partly on whether “it” relates more strongly to “trophy” or “box.” A simplified attention calculation assigns those two tokens weights of 0.39 and 0.61, respectively, producing a contextual vector of [0.39, 0.61]. These values are illustrative, not outputs from a trained model; real transformers use larger representations, multiple heads, and stacked layers. section 12.2.1

