A token is a discrete piece of text that a language model can read or generate. It may be a common word, a word fragment, a character, a symbol, or a special control code. For example, “hungryish” can be assembled from “hungry” and “ish,” even when the complete word is absent from the vocabulary. section 1.8 section 1.9
The token pipeline works in a few practical steps:
- A tokenizer subdivides raw text into valid vocabulary items, roughly favoring the largest available pieces. section 1.9
- The wrapper converts those tokens into integer IDs representing their positions in the model’s vocabulary. section 1.9
- The language model predicts a likely next token, and the wrapper repeatedly feeds that output back to generate a complete response. section 1.9 section 1.34
- The vocabulary balances useful word-sized tokens against smaller pieces; making it contain every word and derivative would make it unnecessarily huge. section 1.12
One prompt, several tokens
The prompt “The dog’s bark was barely” becomes “The | dog | ’s | bark | was | barely,” then the model may append “audible.” It continues predicting tokens until it produces a special end-of-text token. section 1.9
We| the| People| of| the| United| States|,| in| Order| to| form| a| more| perfect| Union|,| establish| Justice|,| insure| domestic| Tran|qu|ility|,| provide| for| the| common| defense|,| promote| the| general| Welfare|,| and| secure| the| Bless|ings| of| Liberty| to| ourselves| and| our| Poster|ity|,| do| ord|ain| and| establish| this| Constitution| for| the| United| States| of| America|.
Compare English and French tokenization directly: familiar English words often remain intact, while French words are split into less meaningful pieces. section 1.11
Tokens are also a practical cost and fairness issue. APIs commonly charge for input and output tokens, and retaining a longer chat history increases the input cost. English-optimized vocabularies may split French or non-Latin text into many more tokens, making those prompts more expensive and sometimes harder to contextualize. section 1.10 section 1.11
