A language model’s vocabulary is the set of tokens it can represent and choose from. Text is split into tokens, which are mapped to integer IDs. During generation, the model selects one token at a time, appends it to the context, and chooses again. section 10.2
From words to choices
“Take aspirin today” might be split into three tokens, though a tokenizer may divide words into smaller pieces. At each generation step, the model chooses from its vocabulary; a typical vocabulary contains 30,000 to 150,000 tokens. section 10.2 section 10.3
Figure 10.2 Language generation mapped as a Markov Decision Process.

Shows language generation as a Markov decision process, with choosing a token from the vocabulary as the action. section 10.3
















