liveBook · Manning

Token

As explained in Timeless Algorithms, The Foundational Ideas

A token is a discrete unit of text represented in a model's vocabulary, such as a word, word fragment, or punctuation mark.

A token is a discrete text unit in a model’s vocabulary. It may represent a whole word, part of a word, punctuation, or another recurring text element. Tokenization turns input text into a sequence of integer identifiers; applying the same tokenizer to the same text produces the same identifiers. section 12.2.1

An identifier is an index, not a numerical measure of meaning. It selects a row from a learned embedding matrix, giving the token its initial vector representation. In a transformer, positional information supplements token embeddings so that the model can distinguish sequence order. section 12.2.1 section 12.1.2

Self-attention makes token representations sensitive to context. For each token, the model weights information from other relevant tokens and combines it into a new representation. Tokens far apart can interact directly rather than relying on information carried through every intervening recurrent state. The resulting representations support a probability distribution over possible next tokens. section 12.1.2 section 12.2.1

A pronoun in context

In “The trophy didn’t fit in the box because it was too…”, a continuation depends partly on whether “it” relates more strongly to “trophy” or “box.” A simplified attention calculation assigns those two tokens weights of 0.39 and 0.61, respectively, producing a contextual vector of [0.39, 0.61]. These values are illustrative, not outputs from a trained model; real transformers use larger representations, multiple heads, and stacked layers. section 12.2.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains identifiers, embeddings, and contextual attention with an illustrative calculation, while simplifying the larger representations and layered operations of real transformers.
Examples
Works through a trophy-and-box pronoun example using two-dimensional vectors and illustrative attention weights.
What you can do after
Evaluate how contextual relationships among tokens can influence a next-token prediction without explicitly labeling a pronoun’s referent.
How it connects
Connects tokenization and embeddings to positional information, self-attention, and probability distributions.

Also explained in

Compare how 2 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.