A vocabulary is the finalized collection of tokens a language model can represent. Before training, an automated tokenization process builds this collection from text, then assigns every token an arbitrary numeric ID. section 2.1.1
Byte Pair Encoding (BPE) builds a vocabulary in simple stages:
- Start with individual characters as tokens. section 2.1.1
- Find neighboring token pairs that occur most often and merge them. section 2.1.1
- Repeat the merges until the vocabulary reaches its target size, typically 50,000 to 100,000 tokens. section 2.1.1
- Assign each finalized token a numeric index; the number identifies its position but carries no meaning by itself. section 2.1.1
One word, several tokens
A common word such as “the” may become one token, while “unbelievable” may split into “un”, “believ”, and “able.” Humans see one word; the model receives three tokens and three numbers. section 2.1.1
Figure 2.1 The same word, seen in two ways. A human reads "unbelievable" as a single unit of meaning. A model never sees the word at all: it receives three tokens and three numbers.

Compare the human view of “unbelievable” with the model’s token-and-number representation. section 2.1.1
















