A token is a discrete unit of text processed by a language model. It can be a whole word, part of a word, punctuation, a space-like marker, or a symbol. section 10.1 section 10.2
A possible token split
“Take aspirin today” might become three tokens: “Take”, “aspirin”, and “today”. Real tokenizers may split some words into smaller pieces, and each token is mapped to an integer ID. section 10.2
When generating language, the model chooses one token at a time from its vocabulary. The prompt and all previously generated tokens form the current state; adding a token extends that sequence. section 10.1 section 10.3
Figure 10.1 Overview of large language models architecture.

The architecture diagram situates tokenization between raw text and the model’s next-token prediction. section 10.2







































