A token is a small text unit that an LLM reads and produces. Tokenization splits language into words, parts of words, or punctuation marks, so a single complex word may become several tokens. section 1.5
After splitting text, the model converts each token into numbers called an embedding. These numerical representations become the atomic units used in computation; tokens with related meanings can occupy nearby positions in the embedding space. section 1.5 section 1.6
Completing a sentence one token at a time
For the prompt “The capital of Italy is ...”, the model uses the token representations and their context to predict a likely next token, such as “Rome.” Repeating that process builds longer text, including sentences, paragraphs, or reports. section 1.5 section 1.7
Figure 1.7 Words are numbers in the eyes of an LLM.
Visualizes how an LLM represents words as numerical coordinates rather than treating them as ordinary text. section 1.5
